Oct 30 21:22:19.226219 ip-10-0-143-61 systemd[1]: Starting Kubernetes Kubelet... Oct 30 21:22:19.572122 ip-10-0-143-61 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:22:19.572122 ip-10-0-143-61 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 21:22:19.572122 ip-10-0-143-61 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:22:19.572122 ip-10-0-143-61 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:22:19.577220 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577122 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:22:19.577220 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577219 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:22:19.577220 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577224 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:22:19.577220 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577227 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:22:19.577343 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577231 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:22:19.577343 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577234 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:22:19.577343 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577238 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:22:19.577343 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577241 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:22:19.577343 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577245 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:22:19.577343 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577247 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:22:19.577343 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577250 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:22:19.577343 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577253 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:22:19.577343 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577256 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:22:19.577343 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577258 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:22:19.577343 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577261 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:22:19.577343 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577263 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:22:19.577343 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577266 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:22:19.577343 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577268 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:22:19.577343 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577271 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:22:19.577343 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577274 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:22:19.577343 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577276 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:22:19.577343 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577279 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:22:19.577343 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577281 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:22:19.577782 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577284 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:22:19.577782 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577287 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:22:19.577782 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577290 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:22:19.577782 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577292 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:22:19.577782 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577295 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:22:19.577782 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577305 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:22:19.577782 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577308 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:22:19.577782 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577310 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:22:19.577782 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577313 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:22:19.577782 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577316 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:22:19.577782 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577318 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:22:19.577782 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577321 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:22:19.577782 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577323 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:22:19.577782 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577326 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:22:19.577782 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577330 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:22:19.577782 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577333 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:22:19.577782 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577335 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:22:19.577782 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577338 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:22:19.577782 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577341 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:22:19.577782 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577344 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:22:19.578275 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577347 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:22:19.578275 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577350 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:22:19.578275 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577353 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:22:19.578275 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577356 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:22:19.578275 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577358 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:22:19.578275 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577361 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:22:19.578275 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577364 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:22:19.578275 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577367 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:22:19.578275 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577369 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:22:19.578275 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577372 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:22:19.578275 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577374 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:22:19.578275 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577377 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:22:19.578275 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577379 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:22:19.578275 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577382 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:22:19.578275 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577384 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:22:19.578275 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577387 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:22:19.578275 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577389 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:22:19.578275 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577393 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:22:19.578275 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577396 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:22:19.578275 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577398 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:22:19.578751 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577401 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:22:19.578751 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577403 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:22:19.578751 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577406 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:22:19.578751 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577408 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:22:19.578751 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577411 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:22:19.578751 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577413 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:22:19.578751 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577417 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:22:19.578751 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577419 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:22:19.578751 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577422 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:22:19.578751 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577425 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:22:19.578751 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577427 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:22:19.578751 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577429 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:22:19.578751 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577432 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:22:19.578751 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577436 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:22:19.578751 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577441 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:22:19.578751 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577444 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:22:19.578751 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577447 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:22:19.578751 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577450 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:22:19.578751 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577453 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:22:19.579224 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577455 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:22:19.579224 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577458 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:22:19.579224 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577461 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:22:19.579224 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577463 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:22:19.579224 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577466 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:22:19.579224 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577468 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:22:19.579224 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577471 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:22:19.579224 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577474 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:22:19.579224 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577477 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:22:19.579224 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577479 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:22:19.579224 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577482 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:22:19.579224 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577484 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:22:19.579224 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577487 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:22:19.579224 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577490 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:22:19.579224 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577492 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:22:19.579224 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577495 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:22:19.579224 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577498 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:22:19.579224 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577500 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:22:19.579224 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577503 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:22:19.579718 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577507 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:22:19.579718 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577510 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:22:19.579718 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577512 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:22:19.579718 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577517 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:22:19.579718 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.577520 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:22:19.579718 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578359 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:22:19.579718 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578367 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:22:19.579718 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578370 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:22:19.579718 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578373 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:22:19.579718 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578376 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:22:19.579718 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578378 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:22:19.579718 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578381 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:22:19.579718 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578384 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:22:19.579718 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578387 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:22:19.579718 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578390 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:22:19.579718 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578392 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:22:19.579718 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578395 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:22:19.579718 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578398 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:22:19.579718 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578400 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:22:19.580199 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578403 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:22:19.580199 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578406 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:22:19.580199 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578408 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:22:19.580199 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578411 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:22:19.580199 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578414 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:22:19.580199 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578416 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:22:19.580199 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578419 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:22:19.580199 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578422 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:22:19.580199 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578424 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:22:19.580199 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578428 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:22:19.580199 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578432 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:22:19.580199 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578435 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:22:19.580199 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578439 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:22:19.580199 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578443 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:22:19.580199 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578446 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:22:19.580199 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578448 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:22:19.580199 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578451 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:22:19.580199 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578453 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:22:19.580199 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578456 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:22:19.580686 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578459 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:22:19.580686 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578461 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:22:19.580686 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578464 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:22:19.580686 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578467 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:22:19.580686 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578469 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:22:19.580686 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578472 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:22:19.580686 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578475 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:22:19.580686 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578486 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:22:19.580686 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578489 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:22:19.580686 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578492 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:22:19.580686 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578494 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:22:19.580686 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578497 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:22:19.580686 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578499 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:22:19.580686 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578502 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:22:19.580686 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578505 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:22:19.580686 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578507 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:22:19.580686 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578510 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:22:19.580686 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578512 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:22:19.580686 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578514 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:22:19.580686 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578517 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:22:19.581177 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578520 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:22:19.581177 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578522 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:22:19.581177 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578525 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:22:19.581177 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578527 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:22:19.581177 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578530 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:22:19.581177 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578533 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:22:19.581177 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578536 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:22:19.581177 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578539 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:22:19.581177 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578541 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:22:19.581177 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578544 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:22:19.581177 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578547 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:22:19.581177 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578549 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:22:19.581177 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578552 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:22:19.581177 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578554 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:22:19.581177 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578557 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:22:19.581177 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578560 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:22:19.581177 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578562 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:22:19.581177 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578565 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:22:19.581177 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578568 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:22:19.581177 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578570 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:22:19.581659 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578573 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:22:19.581659 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578576 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:22:19.581659 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578578 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:22:19.581659 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578581 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:22:19.581659 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578583 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:22:19.581659 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578585 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:22:19.581659 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578588 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:22:19.581659 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578591 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:22:19.581659 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578593 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:22:19.581659 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578595 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:22:19.581659 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578598 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:22:19.581659 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578601 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:22:19.581659 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578603 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:22:19.581659 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578605 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:22:19.581659 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578608 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:22:19.581659 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578611 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:22:19.581659 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578614 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:22:19.581659 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578616 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:22:19.581659 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578619 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:22:19.581659 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578622 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:22:19.582212 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578626 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:22:19.582212 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578630 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:22:19.582212 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578633 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:22:19.582212 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578637 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:22:19.582212 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578639 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:22:19.582212 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578642 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:22:19.582212 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578645 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:22:19.582212 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578647 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:22:19.582212 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578650 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:22:19.582212 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578652 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:22:19.582212 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578655 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:22:19.582212 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578657 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:22:19.582212 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.578660 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:22:19.582212 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579189 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:22:19.582212 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579195 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:22:19.582212 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579198 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:22:19.582212 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579202 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:22:19.582212 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579205 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:22:19.582212 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579208 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:22:19.582657 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579211 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:22:19.582657 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579215 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:22:19.582657 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579219 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:22:19.582657 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579221 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:22:19.582657 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579225 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:22:19.582657 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579228 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:22:19.582657 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579231 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:22:19.582657 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579234 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:22:19.582657 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579238 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:22:19.582657 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579242 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:22:19.582657 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579245 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:22:19.582657 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579249 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:22:19.582657 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579252 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:22:19.582657 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579255 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:22:19.582657 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579258 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:22:19.582657 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579260 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:22:19.582657 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579263 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:22:19.582657 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579265 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:22:19.582657 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579268 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:22:19.583186 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579270 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:22:19.583186 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579273 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:22:19.583186 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579275 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:22:19.583186 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579278 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:22:19.583186 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579280 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:22:19.583186 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579283 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:22:19.583186 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579286 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:22:19.583186 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579289 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:22:19.583186 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579291 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:22:19.583186 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579294 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:22:19.583186 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579296 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:22:19.583186 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579299 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:22:19.583186 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579301 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:22:19.583186 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579313 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:22:19.583186 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579316 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:22:19.583186 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579318 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:22:19.583186 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579321 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:22:19.583186 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579323 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:22:19.583186 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579326 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:22:19.583186 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579328 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:22:19.583694 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579331 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:22:19.583694 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579334 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:22:19.583694 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579337 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:22:19.583694 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579339 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:22:19.583694 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579342 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:22:19.583694 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579345 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:22:19.583694 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579348 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:22:19.583694 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579351 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:22:19.583694 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579353 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:22:19.583694 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579356 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:22:19.583694 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579359 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:22:19.583694 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579361 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:22:19.583694 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579364 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:22:19.583694 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579367 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:22:19.583694 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579369 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:22:19.583694 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579372 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:22:19.583694 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579374 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:22:19.583694 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579377 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:22:19.583694 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579379 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:22:19.583694 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579382 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:22:19.584181 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579385 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:22:19.584181 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579388 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:22:19.584181 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579390 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:22:19.584181 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579393 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:22:19.584181 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579395 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:22:19.584181 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579398 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:22:19.584181 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579400 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:22:19.584181 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579403 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:22:19.584181 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579405 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:22:19.584181 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579408 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:22:19.584181 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579410 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:22:19.584181 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579413 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:22:19.584181 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579415 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:22:19.584181 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579418 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:22:19.584181 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579421 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:22:19.584181 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579424 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:22:19.584181 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579427 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:22:19.584181 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579430 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:22:19.584181 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579433 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:22:19.584624 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579435 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:22:19.584624 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579438 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:22:19.584624 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579440 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:22:19.584624 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579443 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:22:19.584624 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579446 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:22:19.584624 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579448 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:22:19.584624 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579451 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:22:19.584624 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579453 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:22:19.584624 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579456 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:22:19.584624 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579458 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:22:19.584624 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579462 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:22:19.584624 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579464 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:22:19.584624 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579467 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:22:19.584624 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579469 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:22:19.584624 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579472 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:22:19.584624 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579475 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:22:19.584624 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579477 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:22:19.584624 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579480 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:22:19.584624 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579482 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:22:19.585089 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579485 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:22:19.585089 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579487 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:22:19.585089 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.579489 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:22:19.585089 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580520 2582 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 21:22:19.585089 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580536 2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 21:22:19.585089 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580543 2582 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 21:22:19.585089 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580548 2582 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 21:22:19.585089 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580553 2582 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 21:22:19.585089 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580559 2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 21:22:19.585089 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580563 2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 21:22:19.585089 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580568 2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 21:22:19.585089 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580572 2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 21:22:19.585089 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580575 2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 21:22:19.585089 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580578 2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 21:22:19.585089 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580582 2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 21:22:19.585089 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580585 2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 21:22:19.585089 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580588 2582 flags.go:64] FLAG: --cgroup-root="" Oct 30 21:22:19.585089 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580591 2582 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 21:22:19.585089 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580594 2582 flags.go:64] FLAG: --client-ca-file="" Oct 30 21:22:19.585089 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580597 2582 flags.go:64] FLAG: --cloud-provider="external" Oct 30 21:22:19.585089 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580600 2582 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 21:22:19.585089 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580606 2582 flags.go:64] FLAG: --cluster-domain="" Oct 30 21:22:19.585089 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580609 2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 21:22:19.585089 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580613 2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 21:22:19.585651 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580616 2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 21:22:19.585651 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580620 2582 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 21:22:19.585651 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580624 2582 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 21:22:19.585651 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580627 2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 21:22:19.585651 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580631 2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 21:22:19.585651 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580634 2582 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 21:22:19.585651 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580637 2582 flags.go:64] FLAG: --contention-profiling="false" Oct 30 21:22:19.585651 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580640 2582 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 21:22:19.585651 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580644 2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 21:22:19.585651 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580647 2582 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 21:22:19.585651 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580650 2582 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 21:22:19.585651 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580654 2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 21:22:19.585651 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580657 2582 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 21:22:19.585651 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580660 2582 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 21:22:19.585651 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580663 2582 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 21:22:19.585651 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580668 2582 flags.go:64] FLAG: --enable-server="true" Oct 30 21:22:19.585651 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580673 2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 21:22:19.585651 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580677 2582 flags.go:64] FLAG: --event-burst="100" Oct 30 21:22:19.585651 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580681 2582 flags.go:64] FLAG: --event-qps="50" Oct 30 21:22:19.585651 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580684 2582 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 21:22:19.585651 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580687 2582 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 21:22:19.585651 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580691 2582 flags.go:64] FLAG: --eviction-hard="" Oct 30 21:22:19.585651 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580695 2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 21:22:19.585651 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580699 2582 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 21:22:19.586294 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580702 2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 21:22:19.586294 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580705 2582 flags.go:64] FLAG: --eviction-soft="" Oct 30 21:22:19.586294 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580709 2582 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 21:22:19.586294 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580711 2582 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 21:22:19.586294 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580715 2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 21:22:19.586294 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580719 2582 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 21:22:19.586294 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580721 2582 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 21:22:19.586294 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580724 2582 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 21:22:19.586294 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580727 2582 flags.go:64] FLAG: --feature-gates="" Oct 30 21:22:19.586294 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580732 2582 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 21:22:19.586294 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580735 2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 21:22:19.586294 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580738 2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 21:22:19.586294 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580742 2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 21:22:19.586294 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580746 2582 flags.go:64] FLAG: --healthz-port="10248" Oct 30 21:22:19.586294 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580749 2582 flags.go:64] FLAG: --help="false" Oct 30 21:22:19.586294 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580752 2582 flags.go:64] FLAG: --hostname-override="ip-10-0-143-61.us-west-1.compute.internal" Oct 30 21:22:19.586294 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580756 2582 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 21:22:19.586294 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580759 2582 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 21:22:19.586294 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580762 2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 21:22:19.586294 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580765 2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 21:22:19.586294 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580769 2582 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 21:22:19.586294 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580772 2582 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 21:22:19.586294 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580774 2582 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 21:22:19.586844 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580779 2582 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 21:22:19.586844 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580783 2582 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 21:22:19.586844 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580786 2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 21:22:19.586844 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580789 2582 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 21:22:19.586844 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580792 2582 flags.go:64] FLAG: --kube-reserved="" Oct 30 21:22:19.586844 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580795 2582 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 21:22:19.586844 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580798 2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 21:22:19.586844 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580801 2582 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 21:22:19.586844 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580804 2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 21:22:19.586844 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580807 2582 flags.go:64] FLAG: --lock-file="" Oct 30 21:22:19.586844 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580810 2582 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 21:22:19.586844 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580813 2582 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 21:22:19.586844 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580816 2582 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 21:22:19.586844 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580823 2582 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 21:22:19.586844 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580826 2582 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 21:22:19.586844 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580829 2582 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 21:22:19.586844 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580832 2582 flags.go:64] FLAG: --logging-format="text" Oct 30 21:22:19.586844 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580834 2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 21:22:19.586844 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580838 2582 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 21:22:19.586844 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580841 2582 flags.go:64] FLAG: --manifest-url="" Oct 30 21:22:19.586844 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580844 2582 flags.go:64] FLAG: --manifest-url-header="" Oct 30 21:22:19.586844 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580849 2582 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 21:22:19.586844 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580853 2582 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 21:22:19.586844 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580857 2582 flags.go:64] FLAG: --max-pods="110" Oct 30 21:22:19.586844 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580860 2582 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 21:22:19.587556 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580865 2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 21:22:19.587556 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580868 2582 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 21:22:19.587556 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580871 2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 21:22:19.587556 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580874 2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 21:22:19.587556 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580878 2582 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 21:22:19.587556 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580881 2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 21:22:19.587556 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580888 2582 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 21:22:19.587556 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580904 2582 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 21:22:19.587556 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580909 2582 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 21:22:19.587556 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580912 2582 flags.go:64] FLAG: --pod-cidr="" Oct 30 21:22:19.587556 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580915 2582 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 21:22:19.587556 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580918 2582 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 21:22:19.587556 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580921 2582 flags.go:64] FLAG: --pods-per-core="0" Oct 30 21:22:19.587556 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580925 2582 flags.go:64] FLAG: --port="10250" Oct 30 21:22:19.587556 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580928 2582 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 21:22:19.587556 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580931 2582 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0705c1c65e331f110" Oct 30 21:22:19.587556 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580934 2582 flags.go:64] FLAG: --qos-reserved="" Oct 30 21:22:19.587556 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580937 2582 flags.go:64] FLAG: --read-only-port="10255" Oct 30 21:22:19.587556 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580940 2582 flags.go:64] FLAG: --register-node="true" Oct 30 21:22:19.587556 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580943 2582 flags.go:64] FLAG: --register-with-taints="" Oct 30 21:22:19.587556 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580947 2582 flags.go:64] FLAG: --registry-burst="10" Oct 30 21:22:19.587556 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580950 2582 flags.go:64] FLAG: --registry-qps="5" Oct 30 21:22:19.587556 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580953 2582 flags.go:64] FLAG: --reserved-cpus="" Oct 30 21:22:19.587556 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580956 2582 flags.go:64] FLAG: --reserved-memory="" Oct 30 21:22:19.587556 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580961 2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 21:22:19.587556 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580964 2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 21:22:19.588213 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580967 2582 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 21:22:19.588213 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580969 2582 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 21:22:19.588213 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580972 2582 flags.go:64] FLAG: --runonce="false" Oct 30 21:22:19.588213 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580975 2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 21:22:19.588213 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580979 2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 21:22:19.588213 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580982 2582 flags.go:64] FLAG: --seccomp-default="false" Oct 30 21:22:19.588213 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580985 2582 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 21:22:19.588213 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580988 2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 21:22:19.588213 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580992 2582 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 21:22:19.588213 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580995 2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 21:22:19.588213 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.580998 2582 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 21:22:19.588213 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.581001 2582 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 21:22:19.588213 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.581004 2582 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 21:22:19.588213 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.581008 2582 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 21:22:19.588213 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.581013 2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 21:22:19.588213 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.581016 2582 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 21:22:19.588213 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.581019 2582 flags.go:64] FLAG: --system-cgroups="" Oct 30 21:22:19.588213 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.581022 2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 21:22:19.588213 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.581028 2582 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 21:22:19.588213 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.581031 2582 flags.go:64] FLAG: --tls-cert-file="" Oct 30 21:22:19.588213 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.581034 2582 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 21:22:19.588213 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.581039 2582 flags.go:64] FLAG: --tls-min-version="" Oct 30 21:22:19.588213 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.581042 2582 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 21:22:19.588213 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.581045 2582 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 21:22:19.588213 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.581048 2582 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 21:22:19.588818 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.581051 2582 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 21:22:19.588818 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.581054 2582 flags.go:64] FLAG: --v="2" Oct 30 21:22:19.588818 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.581064 2582 flags.go:64] FLAG: --version="false" Oct 30 21:22:19.588818 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.581068 2582 flags.go:64] FLAG: --vmodule="" Oct 30 21:22:19.588818 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.581072 2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 21:22:19.588818 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.581076 2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 21:22:19.588818 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581237 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:22:19.588818 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581242 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:22:19.588818 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581246 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:22:19.588818 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581249 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:22:19.588818 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581253 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:22:19.588818 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581256 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:22:19.588818 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581258 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:22:19.588818 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581262 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:22:19.588818 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581265 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:22:19.588818 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581267 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:22:19.588818 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581270 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:22:19.588818 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581273 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:22:19.588818 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581275 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:22:19.588818 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581278 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:22:19.588818 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581281 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:22:19.589408 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581285 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:22:19.589408 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581287 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:22:19.589408 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581290 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:22:19.589408 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581293 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:22:19.589408 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581297 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:22:19.589408 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581301 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:22:19.589408 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581304 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:22:19.589408 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581308 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:22:19.589408 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581311 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:22:19.589408 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581313 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:22:19.589408 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581316 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:22:19.589408 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581319 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:22:19.589408 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581322 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:22:19.589408 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581325 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:22:19.589408 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581327 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:22:19.589408 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581330 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:22:19.589408 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581332 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:22:19.589408 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581335 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:22:19.589408 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581337 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:22:19.589408 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581340 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:22:19.589883 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581342 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:22:19.589883 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581345 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:22:19.589883 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581347 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:22:19.589883 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581350 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:22:19.589883 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581352 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:22:19.589883 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581356 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:22:19.589883 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581360 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:22:19.589883 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581363 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:22:19.589883 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581366 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:22:19.589883 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581368 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:22:19.589883 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581371 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:22:19.589883 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581375 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:22:19.589883 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581378 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:22:19.589883 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581380 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:22:19.589883 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581383 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:22:19.589883 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581386 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:22:19.589883 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581389 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:22:19.589883 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581391 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:22:19.589883 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581394 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:22:19.590356 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581396 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:22:19.590356 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581399 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:22:19.590356 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581402 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:22:19.590356 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581404 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:22:19.590356 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581407 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:22:19.590356 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581409 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:22:19.590356 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581412 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:22:19.590356 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581415 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:22:19.590356 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581417 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:22:19.590356 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581420 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:22:19.590356 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581423 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:22:19.590356 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581426 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:22:19.590356 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581428 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:22:19.590356 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581431 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:22:19.590356 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581434 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:22:19.590356 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581436 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:22:19.590356 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581439 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:22:19.590356 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581441 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:22:19.590356 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581444 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:22:19.590356 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581447 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:22:19.590839 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581449 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:22:19.590839 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581452 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:22:19.590839 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581455 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:22:19.590839 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581457 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:22:19.590839 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581461 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:22:19.590839 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581464 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:22:19.590839 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581466 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:22:19.590839 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581469 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:22:19.590839 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581472 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:22:19.590839 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581475 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:22:19.590839 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581477 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:22:19.590839 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581480 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:22:19.590839 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581482 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:22:19.590839 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581485 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:22:19.590839 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581488 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:22:19.590839 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581490 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:22:19.590839 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581493 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:22:19.590839 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581495 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:22:19.590839 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581498 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:22:19.591319 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581500 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:22:19.591319 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581503 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:22:19.591319 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581506 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:22:19.591319 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581508 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:22:19.591319 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581511 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:22:19.591319 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581514 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:22:19.591319 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581516 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:22:19.591319 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581519 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:22:19.591319 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581521 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:22:19.591319 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581524 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:22:19.591319 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581527 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:22:19.591319 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581530 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:22:19.591319 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581532 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:22:19.591319 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.581558 2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:22:19.591319 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581594 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:22:19.591319 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581599 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:22:19.591705 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581602 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:22:19.591705 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581604 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:22:19.591705 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581608 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:22:19.591705 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581611 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:22:19.591705 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581614 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:22:19.591705 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581617 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:22:19.591705 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581619 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:22:19.591705 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581622 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:22:19.591705 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581625 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:22:19.591705 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581627 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:22:19.591705 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581631 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:22:19.591705 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581635 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:22:19.591705 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581638 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:22:19.591705 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581641 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:22:19.591705 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581644 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:22:19.591705 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581647 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:22:19.591705 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581649 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:22:19.591705 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581652 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:22:19.591705 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581655 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:22:19.591705 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581657 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:22:19.592275 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581660 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:22:19.592275 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581663 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:22:19.592275 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581665 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:22:19.592275 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581668 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:22:19.592275 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581671 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:22:19.592275 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581673 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:22:19.592275 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581676 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:22:19.592275 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581679 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:22:19.592275 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581681 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:22:19.592275 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581684 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:22:19.592275 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581687 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:22:19.592275 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581690 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:22:19.592275 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581694 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:22:19.592275 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581697 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:22:19.592275 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581702 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:22:19.592275 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581705 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:22:19.592275 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581708 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:22:19.592275 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581710 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:22:19.592275 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581713 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:22:19.592756 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581715 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:22:19.592756 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581718 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:22:19.592756 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581721 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:22:19.592756 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581723 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:22:19.592756 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581726 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:22:19.592756 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581728 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:22:19.592756 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581731 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:22:19.592756 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581733 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:22:19.592756 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581736 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:22:19.592756 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581739 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:22:19.592756 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581741 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:22:19.592756 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581744 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:22:19.592756 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581746 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:22:19.592756 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581749 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:22:19.592756 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581751 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:22:19.592756 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581756 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:22:19.592756 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581758 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:22:19.592756 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581761 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:22:19.592756 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581763 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:22:19.592756 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581766 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:22:19.593237 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581768 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:22:19.593237 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581771 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:22:19.593237 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581774 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:22:19.593237 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581776 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:22:19.593237 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581779 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:22:19.593237 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581782 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:22:19.593237 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581785 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:22:19.593237 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581789 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:22:19.593237 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581791 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:22:19.593237 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581794 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:22:19.593237 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581796 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:22:19.593237 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581799 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:22:19.593237 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581802 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:22:19.593237 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581804 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:22:19.593237 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581807 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:22:19.593237 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581809 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:22:19.593237 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581812 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:22:19.593237 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581814 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:22:19.593237 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581817 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:22:19.593696 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581819 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:22:19.593696 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581822 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:22:19.593696 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581825 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:22:19.593696 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581827 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:22:19.593696 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581830 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:22:19.593696 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581832 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:22:19.593696 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581835 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:22:19.593696 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581837 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:22:19.593696 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581840 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:22:19.593696 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581842 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:22:19.593696 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581860 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:22:19.593696 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581863 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:22:19.593696 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581866 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:22:19.593696 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581869 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:22:19.593696 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581872 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:22:19.593696 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581875 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:22:19.593696 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581878 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:22:19.593696 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581880 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:22:19.593696 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581884 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:22:19.593696 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581886 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:22:19.594180 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581890 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:22:19.594180 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581905 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:22:19.594180 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581908 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:22:19.594180 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581911 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:22:19.594180 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581914 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:22:19.594180 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.581917 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:22:19.594180 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.589798 2582 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 21:22:19.594180 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.589819 2582 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 21:22:19.594180 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.589838 2582 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 21:22:19.594180 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.589843 2582 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 21:22:19.594180 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.589920 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:22:19.594180 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.589927 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:22:19.594180 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.589931 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:22:19.594180 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.589934 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:22:19.594180 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.589937 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:22:19.594180 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.589940 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:22:19.594180 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.589943 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:22:19.594180 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.589946 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:22:19.594180 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.589949 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:22:19.594632 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.589952 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:22:19.594632 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.589954 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:22:19.594632 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.589957 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:22:19.594632 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.589960 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:22:19.594632 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.589962 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:22:19.594632 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.589965 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:22:19.594632 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.589968 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:22:19.594632 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.589971 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:22:19.594632 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.589974 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:22:19.594632 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.589977 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:22:19.594632 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.589979 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:22:19.594632 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.589983 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:22:19.594632 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.589986 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:22:19.594632 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.589988 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:22:19.594632 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.589991 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:22:19.594632 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.589994 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:22:19.594632 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.589996 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:22:19.594632 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.589999 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:22:19.594632 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590001 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:22:19.594632 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590005 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:22:19.595126 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590008 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:22:19.595126 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590010 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:22:19.595126 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590013 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:22:19.595126 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590016 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:22:19.595126 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590018 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:22:19.595126 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590021 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:22:19.595126 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590024 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:22:19.595126 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590027 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:22:19.595126 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590030 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:22:19.595126 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590032 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:22:19.595126 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590035 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:22:19.595126 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590038 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:22:19.595126 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590040 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:22:19.595126 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590043 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:22:19.595126 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590046 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:22:19.595126 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590048 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:22:19.595126 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590051 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:22:19.595126 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590054 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:22:19.595126 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590057 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:22:19.595576 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590060 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:22:19.595576 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590063 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:22:19.595576 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590066 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:22:19.595576 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590068 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:22:19.595576 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590071 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:22:19.595576 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590074 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:22:19.595576 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590076 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:22:19.595576 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590079 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:22:19.595576 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590082 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:22:19.595576 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590084 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:22:19.595576 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590087 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:22:19.595576 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590090 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:22:19.595576 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590093 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:22:19.595576 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590097 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:22:19.595576 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590100 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:22:19.595576 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590102 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:22:19.595576 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590105 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:22:19.595576 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590109 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:22:19.595576 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590113 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:22:19.595576 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590116 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:22:19.596106 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590119 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:22:19.596106 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590122 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:22:19.596106 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590125 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:22:19.596106 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590128 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:22:19.596106 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590130 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:22:19.596106 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590133 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:22:19.596106 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590136 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:22:19.596106 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590140 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:22:19.596106 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590143 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:22:19.596106 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590146 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:22:19.596106 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590149 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:22:19.596106 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590152 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:22:19.596106 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590155 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:22:19.596106 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590157 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:22:19.596106 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590160 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:22:19.596106 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590163 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:22:19.596106 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590165 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:22:19.596106 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590168 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:22:19.596106 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590171 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:22:19.596537 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590174 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:22:19.596537 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590176 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:22:19.596537 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590179 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:22:19.596537 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590181 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:22:19.596537 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590184 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:22:19.596537 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590187 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:22:19.596537 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590190 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:22:19.596537 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590193 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:22:19.596537 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590196 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:22:19.596537 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590199 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:22:19.596537 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590201 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:22:19.596537 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590204 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:22:19.596537 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590206 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:22:19.596537 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590209 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:22:19.596537 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590211 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:22:19.596537 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590214 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:22:19.596537 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590216 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:22:19.596537 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590219 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:22:19.596537 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590222 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:22:19.596989 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.590252 2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:22:19.596989 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590399 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:22:19.596989 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590405 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:22:19.596989 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590410 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:22:19.596989 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590414 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:22:19.596989 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590416 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:22:19.596989 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590419 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:22:19.596989 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590422 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:22:19.596989 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590425 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:22:19.596989 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590427 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:22:19.596989 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590430 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:22:19.596989 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590432 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:22:19.596989 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590435 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:22:19.596989 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590438 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:22:19.596989 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590441 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:22:19.596989 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590443 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:22:19.597570 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590446 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:22:19.597570 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590448 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:22:19.597570 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590451 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:22:19.597570 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590454 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:22:19.597570 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590457 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:22:19.597570 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590459 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:22:19.597570 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590462 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:22:19.597570 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590464 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:22:19.597570 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590467 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:22:19.597570 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590470 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:22:19.597570 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590473 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:22:19.597570 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590475 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:22:19.597570 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590478 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:22:19.597570 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590481 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:22:19.597570 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590483 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:22:19.597570 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590486 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:22:19.597570 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590489 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:22:19.597570 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590491 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:22:19.597570 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590494 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:22:19.597570 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590497 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:22:19.598245 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590499 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:22:19.598245 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590501 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:22:19.598245 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590504 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:22:19.598245 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590507 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:22:19.598245 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590509 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:22:19.598245 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590512 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:22:19.598245 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590514 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:22:19.598245 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590517 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:22:19.598245 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590519 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:22:19.598245 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590523 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:22:19.598245 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590526 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:22:19.598245 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590529 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:22:19.598245 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590531 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:22:19.598245 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590534 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:22:19.598245 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590537 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:22:19.598245 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590540 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:22:19.598245 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590542 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:22:19.598245 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590546 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:22:19.598245 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590550 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:22:19.598781 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590553 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:22:19.598781 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590556 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:22:19.598781 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590559 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:22:19.598781 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590561 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:22:19.598781 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590564 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:22:19.598781 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590566 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:22:19.598781 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590569 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:22:19.598781 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590572 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:22:19.598781 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590574 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:22:19.598781 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590577 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:22:19.598781 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590579 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:22:19.598781 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590582 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:22:19.598781 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590584 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:22:19.598781 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590587 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:22:19.598781 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590589 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:22:19.598781 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590592 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:22:19.598781 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590594 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:22:19.598781 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590597 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:22:19.598781 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590599 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:22:19.599298 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590602 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:22:19.599298 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590604 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:22:19.599298 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590607 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:22:19.599298 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590609 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:22:19.599298 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590612 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:22:19.599298 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590615 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:22:19.599298 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590617 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:22:19.599298 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590620 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:22:19.599298 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590622 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:22:19.599298 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590625 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:22:19.599298 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590628 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:22:19.599298 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590630 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:22:19.599298 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590633 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:22:19.599298 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590636 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:22:19.599298 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590638 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:22:19.599298 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590641 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:22:19.599298 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590643 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:22:19.599298 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590645 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:22:19.599298 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590648 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:22:19.599809 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590651 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:22:19.599809 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590653 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:22:19.599809 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590656 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:22:19.599809 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590658 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:22:19.599809 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590661 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:22:19.599809 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590663 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:22:19.599809 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590666 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:22:19.599809 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590668 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:22:19.599809 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590671 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:22:19.599809 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590674 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:22:19.599809 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590676 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:22:19.599809 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590679 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:22:19.599809 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590681 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:22:19.599809 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590684 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:22:19.599809 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.590707 2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:22:19.599809 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590742 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:22:19.600224 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590745 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:22:19.600224 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590748 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:22:19.600224 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590751 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:22:19.600224 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590754 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:22:19.600224 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590757 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:22:19.600224 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590759 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:22:19.600224 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590762 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:22:19.600224 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590768 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:22:19.600224 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590770 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:22:19.600224 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590773 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:22:19.600224 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590776 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:22:19.600224 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590779 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:22:19.600224 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590782 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:22:19.600224 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590785 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:22:19.600224 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590787 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:22:19.600224 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590790 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:22:19.600224 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590793 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:22:19.600224 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590796 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:22:19.600224 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590799 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:22:19.600224 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590801 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:22:19.600707 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590804 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:22:19.600707 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590807 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:22:19.600707 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590809 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:22:19.600707 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590812 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:22:19.600707 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590815 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:22:19.600707 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590817 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:22:19.600707 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590820 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:22:19.600707 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590822 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:22:19.600707 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590825 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:22:19.600707 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590827 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:22:19.600707 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590830 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:22:19.600707 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590833 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:22:19.600707 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590837 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:22:19.600707 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590841 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:22:19.600707 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590844 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:22:19.600707 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590846 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:22:19.600707 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590850 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:22:19.600707 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590853 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:22:19.600707 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590856 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:22:19.601250 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590860 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:22:19.601250 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590863 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:22:19.601250 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590866 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:22:19.601250 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590869 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:22:19.601250 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590872 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:22:19.601250 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590874 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:22:19.601250 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590877 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:22:19.601250 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590880 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:22:19.601250 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590882 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:22:19.601250 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590885 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:22:19.601250 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590887 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:22:19.601250 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590890 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:22:19.601250 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590892 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:22:19.601250 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590908 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:22:19.601250 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590911 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:22:19.601250 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590914 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:22:19.601250 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590916 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:22:19.601250 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590919 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:22:19.601250 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590922 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:22:19.601717 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590924 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:22:19.601717 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590927 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:22:19.601717 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590929 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:22:19.601717 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590932 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:22:19.601717 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590935 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:22:19.601717 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590937 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:22:19.601717 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590940 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:22:19.601717 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590943 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:22:19.601717 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590945 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:22:19.601717 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590948 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:22:19.601717 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590950 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:22:19.601717 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590954 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:22:19.601717 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590957 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:22:19.601717 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590961 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:22:19.601717 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590964 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:22:19.601717 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590967 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:22:19.601717 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590969 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:22:19.601717 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590972 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:22:19.601717 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590974 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:22:19.602353 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590977 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:22:19.602353 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590980 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:22:19.602353 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590982 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:22:19.602353 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590985 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:22:19.602353 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590987 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:22:19.602353 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590990 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:22:19.602353 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590992 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:22:19.602353 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590995 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:22:19.602353 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.590997 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:22:19.602353 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.591001 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:22:19.602353 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.591007 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:22:19.602353 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.591010 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:22:19.602353 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.591013 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:22:19.602353 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.591016 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:22:19.602353 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.591019 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:22:19.602353 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.591022 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:22:19.602353 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.591025 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:22:19.602353 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.591027 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:22:19.602353 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.591030 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:22:19.602353 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.591032 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:22:19.602869 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.591035 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:22:19.602869 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.591037 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:22:19.602869 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.591040 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:22:19.602869 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.591042 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:22:19.602869 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.591045 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:22:19.602869 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.591048 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:22:19.602869 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.591052 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:22:19.602869 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:19.591054 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:22:19.602869 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.591132 2582 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 21:22:19.602869 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.594317 2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 21:22:19.602869 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.595240 2582 server.go:1014] "Starting client certificate rotation" Oct 30 21:22:19.602869 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.595338 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:22:19.602869 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.597726 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:22:19.602869 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.599063 2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:22:19.603436 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.603119 2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:22:19.613217 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.613197 2582 log.go:25] "Validated CRI v1 runtime API" Oct 30 21:22:19.617854 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.617838 2582 log.go:25] "Validated CRI v1 image API" Oct 30 21:22:19.618677 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.618654 2582 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 21:22:19.622187 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.622166 2582 fs.go:135] Filesystem UUIDs: map[509c864e-1681-4cb1-a673-5aa85d4d34b0:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 f2d395fa-97f1-4864-b1e7-a627def2f004:/dev/nvme0n1p4] Oct 30 21:22:19.622262 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.622188 2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 21:22:19.627478 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.627357 2582 manager.go:217] Machine: {Timestamp:2026-10-30 21:22:19.625965993 +0000 UTC m=+0.300134869 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099612 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b738886855f32b4f9f8b105bc9374 SystemUUID:ec2b7388-8685-5f32-b4f9-f8b105bc9374 BootID:2071c5e8-f6a6-40fc-a852-71159816c8d8 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:68:6d:87:0b:2d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:68:6d:87:0b:2d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f2:aa:c7:d3:cd:8f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 21:22:19.627478 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.627469 2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 21:22:19.627657 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.627583 2582 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 21:22:19.630843 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.630816 2582 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 21:22:19.630996 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.630844 2582 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-61.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 21:22:19.631050 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.630999 2582 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 21:22:19.631050 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.631015 2582 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 21:22:19.631050 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.631048 2582 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:22:19.631127 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.631067 2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:22:19.632286 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.632275 2582 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 21:22:19.636522 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.636504 2582 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 21:22:19.636656 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.636646 2582 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 21:22:19.637419 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.637407 2582 kubelet.go:493] "Attempting to sync node with API server" Oct 30 21:22:19.637463 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.637423 2582 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 21:22:19.637463 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.637435 2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 21:22:19.637463 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.637444 2582 kubelet.go:405] "Adding apiserver pod source" Oct 30 21:22:19.637463 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.637452 2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 21:22:19.637849 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.637836 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:22:19.640323 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.640307 2582 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 21:22:19.647577 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.647561 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:22:19.648535 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.648523 2582 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 21:22:19.648583 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.648554 2582 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 21:22:19.648726 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.648717 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 21:22:19.648754 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.648732 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 21:22:19.648754 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.648739 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 21:22:19.648754 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.648751 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 21:22:19.648827 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.648768 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 21:22:19.648827 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.648774 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 21:22:19.649441 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.649432 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 21:22:19.649470 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.649442 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 21:22:19.649470 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.649449 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 21:22:19.649470 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.649455 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 21:22:19.649470 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.649469 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 21:22:19.649595 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.649478 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 21:22:19.650237 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.650229 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 21:22:19.650267 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.650238 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 21:22:19.655730 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.655707 2582 server.go:1263] "Started kubelet" Oct 30 21:22:19.655857 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.655783 2582 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 21:22:19.655857 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.655808 2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 21:22:19.655975 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.655883 2582 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 21:22:19.656485 ip-10-0-143-61 systemd[1]: Started Kubernetes Kubelet. Oct 30 21:22:19.657513 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.657494 2582 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 21:22:19.657727 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.657711 2582 server.go:317] "Adding debug handlers to kubelet server" Oct 30 21:22:19.662195 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:19.662169 2582 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 21:22:19.666015 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.665996 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 21:22:19.666604 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.666589 2582 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 21:22:19.666918 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:19.666884 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-61.us-west-1.compute.internal\" not found" Oct 30 21:22:19.667450 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.667430 2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 21:22:19.667539 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.667432 2582 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 21:22:19.667539 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.667468 2582 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 21:22:19.667539 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.667525 2582 factory.go:55] Registering systemd factory Oct 30 21:22:19.667640 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.667549 2582 reconstruct.go:97] "Volume reconstruction finished" Oct 30 21:22:19.667640 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.667559 2582 reconciler.go:29] "Reconciler: start to sync state" Oct 30 21:22:19.667640 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.667564 2582 factory.go:223] Registration of the systemd container factory successfully Oct 30 21:22:19.667824 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.667802 2582 factory.go:153] Registering CRI-O factory Oct 30 21:22:19.667824 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.667825 2582 factory.go:223] Registration of the crio container factory successfully Oct 30 21:22:19.667979 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.667890 2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 21:22:19.667979 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.667967 2582 factory.go:103] Registering Raw factory Oct 30 21:22:19.668093 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.667988 2582 manager.go:1199] Started watching for new ooms in manager Oct 30 21:22:19.669017 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.669003 2582 manager.go:319] Starting recovery of all containers Oct 30 21:22:19.679388 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.679370 2582 manager.go:324] Recovery completed Oct 30 21:22:19.685308 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.685292 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:22:19.686824 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.686807 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-61.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:22:19.686914 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.686837 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-61.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:22:19.686914 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.686848 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-61.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:22:19.687429 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.687409 2582 cpu_manager.go:226] "Starting" policy="none" Oct 30 21:22:19.687429 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.687423 2582 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 21:22:19.687542 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.687438 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:22:19.687542 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.687443 2582 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 21:22:19.688119 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.688102 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-61.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:22:19.688213 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.688127 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-61.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:22:19.688213 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.688137 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-61.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:22:19.689624 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.689611 2582 policy_none.go:50] "Start" Oct 30 21:22:19.689679 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.689627 2582 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 21:22:19.689679 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.689640 2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 21:22:19.691555 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.691543 2582 policy_none.go:44] "Start" Oct 30 21:22:19.725635 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.725587 2582 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 21:22:19.728268 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.728254 2582 manager.go:341] "Starting Device Plugin manager" Oct 30 21:22:19.728358 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:19.728343 2582 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 21:22:19.728404 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.728360 2582 server.go:87] "Starting device plugin registration server" Oct 30 21:22:19.728662 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.728649 2582 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 21:22:19.728703 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.728666 2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 21:22:19.729162 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.728857 2582 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 21:22:19.729162 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.728949 2582 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 21:22:19.729162 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.728957 2582 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 21:22:19.729430 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:19.729412 2582 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 21:22:19.729486 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:19.729478 2582 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-61.us-west-1.compute.internal\" not found" Oct 30 21:22:19.788970 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.788934 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 21:22:19.791109 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.790102 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 21:22:19.791109 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.790124 2582 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 21:22:19.791109 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.790146 2582 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 21:22:19.791109 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:19.790183 2582 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 21:22:19.798437 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:19.798408 2582 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-61.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 21:22:19.829682 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.829618 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:22:19.830537 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.830516 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-61.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:22:19.830617 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.830556 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-61.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:22:19.830617 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.830572 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-61.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:22:19.830617 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.830604 2582 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-61.us-west-1.compute.internal" Oct 30 21:22:19.846663 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.846634 2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-61.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 21:22:19.847397 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:19.846671 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-61.us-west-1.compute.internal.18e36b8f42044aa2 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-61.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-61.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 21:22:19.655662242 +0000 UTC m=+0.329831119,LastTimestamp:2026-10-30 21:22:19.655662242 +0000 UTC m=+0.329831119,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-61.us-west-1.compute.internal,}" Oct 30 21:22:19.850304 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:19.850284 2582 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 21:22:19.850363 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:19.850333 2582 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-61.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 21:22:19.877005 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.876981 2582 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 21:22:19.877086 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.877066 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dqngl" Oct 30 21:22:19.891064 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.891032 2582 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-61.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-61.us-west-1.compute.internal"] Oct 30 21:22:19.891137 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.891101 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:22:19.892026 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.892009 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-61.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:22:19.892106 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.892041 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-61.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:22:19.892106 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.892053 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-61.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:22:19.894343 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.894331 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:22:19.895007 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.894980 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-61.us-west-1.compute.internal" Oct 30 21:22:19.895108 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.895019 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:22:19.895108 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.895037 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-61.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:22:19.895108 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.895060 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-61.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:22:19.895108 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.895075 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-61.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:22:19.895683 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.895668 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-61.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:22:19.895767 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.895690 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-61.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:22:19.895767 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.895700 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-61.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:22:19.897185 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.897167 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-61.us-west-1.compute.internal" Oct 30 21:22:19.897282 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.897191 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:22:19.897823 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.897807 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-61.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:22:19.897921 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.897834 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-61.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:22:19.897921 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.897846 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-61.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:22:19.928956 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:19.928933 2582 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-61.us-west-1.compute.internal\" not found" node="ip-10-0-143-61.us-west-1.compute.internal" Oct 30 21:22:19.931632 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:19.931611 2582 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-61.us-west-1.compute.internal\" not found" node="ip-10-0-143-61.us-west-1.compute.internal" Oct 30 21:22:19.959385 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.959357 2582 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:22:19.959522 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.959398 2582 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:22:19.966920 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.966888 2582 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-61.us-west-1.compute.internal" Oct 30 21:22:19.966984 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:19.966924 2582 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-61.us-west-1.compute.internal\": node \"ip-10-0-143-61.us-west-1.compute.internal\" not found" Oct 30 21:22:19.969002 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.968985 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f05411f241c17e725b228582bf16173f-config\") pod \"kube-apiserver-proxy-ip-10-0-143-61.us-west-1.compute.internal\" (UID: \"f05411f241c17e725b228582bf16173f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-61.us-west-1.compute.internal" Oct 30 21:22:19.969047 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.969009 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ddfcdf71a39fda8bf039a6129472bc5e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-61.us-west-1.compute.internal\" (UID: \"ddfcdf71a39fda8bf039a6129472bc5e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-61.us-west-1.compute.internal" Oct 30 21:22:19.969047 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:19.969028 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ddfcdf71a39fda8bf039a6129472bc5e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-61.us-west-1.compute.internal\" (UID: \"ddfcdf71a39fda8bf039a6129472bc5e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-61.us-west-1.compute.internal" Oct 30 21:22:20.070139 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:20.069959 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f05411f241c17e725b228582bf16173f-config\") pod \"kube-apiserver-proxy-ip-10-0-143-61.us-west-1.compute.internal\" (UID: \"f05411f241c17e725b228582bf16173f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-61.us-west-1.compute.internal" Oct 30 21:22:20.070321 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:20.070147 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ddfcdf71a39fda8bf039a6129472bc5e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-61.us-west-1.compute.internal\" (UID: \"ddfcdf71a39fda8bf039a6129472bc5e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-61.us-west-1.compute.internal" Oct 30 21:22:20.070321 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:20.070171 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ddfcdf71a39fda8bf039a6129472bc5e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-61.us-west-1.compute.internal\" (UID: \"ddfcdf71a39fda8bf039a6129472bc5e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-61.us-west-1.compute.internal" Oct 30 21:22:20.070321 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:20.070085 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f05411f241c17e725b228582bf16173f-config\") pod \"kube-apiserver-proxy-ip-10-0-143-61.us-west-1.compute.internal\" (UID: \"f05411f241c17e725b228582bf16173f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-61.us-west-1.compute.internal" Oct 30 21:22:20.070321 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:20.070222 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ddfcdf71a39fda8bf039a6129472bc5e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-61.us-west-1.compute.internal\" (UID: \"ddfcdf71a39fda8bf039a6129472bc5e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-61.us-west-1.compute.internal" Oct 30 21:22:20.070321 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:20.070242 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ddfcdf71a39fda8bf039a6129472bc5e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-61.us-west-1.compute.internal\" (UID: \"ddfcdf71a39fda8bf039a6129472bc5e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-61.us-west-1.compute.internal" Oct 30 21:22:20.091044 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:20.090987 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-61.us-west-1.compute.internal\" not found" Oct 30 21:22:20.191823 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:20.191791 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-61.us-west-1.compute.internal\" not found" Oct 30 21:22:20.232575 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:20.232538 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-61.us-west-1.compute.internal" Oct 30 21:22:20.234679 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:20.234660 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-61.us-west-1.compute.internal" Oct 30 21:22:20.292075 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:20.292036 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-61.us-west-1.compute.internal\" not found" Oct 30 21:22:20.392891 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:20.392868 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-61.us-west-1.compute.internal\" not found" Oct 30 21:22:20.493845 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:20.493807 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-61.us-west-1.compute.internal\" not found" Oct 30 21:22:20.594284 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:20.594244 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-61.us-west-1.compute.internal\" not found" Oct 30 21:22:20.595450 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:20.595416 2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 21:22:20.595616 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:20.595591 2582 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 21:22:20.595744 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:20.595627 2582 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 21:22:20.595818 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:20.595684 2582 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://af1d03f21ae4249649589735f9d6a3af-3cfe10bb520c60ba.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-143-61.us-west-1.compute.internal.18e36b8f43dfc68c\": read tcp 10.0.143.61:60834->18.217.85.158:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-143-61.us-west-1.compute.internal.18e36b8f43dfc68c default 5298 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-61.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-61.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-61.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 21:22:19 +0000 UTC,LastTimestamp:2026-10-30 21:22:19.892025796 +0000 UTC m=+0.566194675,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-61.us-west-1.compute.internal,}" Oct 30 21:22:20.696166 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:20.696075 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-61.us-west-1.compute.internal\" not found" Oct 30 21:22:20.796819 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:20.796784 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-61.us-west-1.compute.internal\" not found" Oct 30 21:22:20.839884 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:20.839852 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:22:20.841019 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:20.841001 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-61.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:22:20.841127 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:20.841032 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-61.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:22:20.841127 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:20.841044 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-61.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:22:20.882155 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:20.882098 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:17:20 +0000 UTC" deadline="2028-04-02 14:05:33.246909404 +0000 UTC" Oct 30 21:22:20.882155 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:20.882149 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12472h43m12.364764465s" Oct 30 21:22:20.897577 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:20.897550 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-61.us-west-1.compute.internal\" not found" Oct 30 21:22:20.997737 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:20.997653 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-61.us-west-1.compute.internal\" not found" Oct 30 21:22:21.099005 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:21.098966 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-61.us-west-1.compute.internal\" not found" Oct 30 21:22:21.199098 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:21.199061 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-61.us-west-1.compute.internal\" not found" Oct 30 21:22:21.233471 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.233442 2582 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:22:21.267608 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.267527 2582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-61.us-west-1.compute.internal" Oct 30 21:22:21.335564 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.335537 2582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:22:21.336889 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.336872 2582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-61.us-west-1.compute.internal" Oct 30 21:22:21.352385 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:21.352351 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddfcdf71a39fda8bf039a6129472bc5e.slice/crio-2e8ccef945c9b2ab26da19f76a836a73756872b7554e0a67985accacdcb37fe9 WatchSource:0}: Error finding container 2e8ccef945c9b2ab26da19f76a836a73756872b7554e0a67985accacdcb37fe9: Status 404 returned error can't find the container with id 2e8ccef945c9b2ab26da19f76a836a73756872b7554e0a67985accacdcb37fe9 Oct 30 21:22:21.358038 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.358022 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:22:21.381246 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:21.381204 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf05411f241c17e725b228582bf16173f.slice/crio-be8bf0429e5f966f001d3467b942c1f2c894b8e25c67bdba0df0bce3b28e9b59 WatchSource:0}: Error finding container be8bf0429e5f966f001d3467b942c1f2c894b8e25c67bdba0df0bce3b28e9b59: Status 404 returned error can't find the container with id be8bf0429e5f966f001d3467b942c1f2c894b8e25c67bdba0df0bce3b28e9b59 Oct 30 21:22:21.411075 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.411040 2582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:22:21.411075 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.411074 2582 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:22:21.640182 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.640158 2582 apiserver.go:52] "Watching apiserver" Oct 30 21:22:21.657207 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.657182 2582 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:22:21.666563 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.666537 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 21:22:21.752352 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.752312 2582 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 21:22:21.752697 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.752674 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-dpr6c","openshift-dns/node-resolver-8rbsm","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-61.us-west-1.compute.internal","openshift-multus/multus-additional-cni-plugins-jtrcb","openshift-multus/network-metrics-daemon-lz8cq","openshift-network-diagnostics/network-check-target-wv464","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvfbv","openshift-multus/multus-n2sfk","openshift-network-operator/iptables-alerter-mxfq6","openshift-image-registry/node-ca-8vqf9","kube-system/global-pull-secret-syncer-6nncz","kube-system/kube-apiserver-proxy-ip-10-0-143-61.us-west-1.compute.internal","openshift-ovn-kubernetes/ovnkube-node-j4v8r","kube-system/konnectivity-agent-75hkd"] Oct 30 21:22:21.756518 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.756494 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvfbv" Oct 30 21:22:21.759032 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.759011 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:21.761280 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.761258 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-75hkd" Oct 30 21:22:21.763702 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.763687 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jtrcb" Oct 30 21:22:21.766149 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.766131 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lz8cq" Oct 30 21:22:21.766250 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:21.766206 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lz8cq" podUID="83a93696-9f3b-472d-9fc8-92350746df75" Oct 30 21:22:21.769258 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.769236 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wv464" Oct 30 21:22:21.769355 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:21.769320 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wv464" podUID="2496f073-7bf0-48af-b091-be6b964117ca" Oct 30 21:22:21.771585 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.771570 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mxfq6" Oct 30 21:22:21.774039 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.774022 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:22:21.776429 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.776413 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8rbsm" Oct 30 21:22:21.778958 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.778942 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8vqf9" Oct 30 21:22:21.779874 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.779857 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-run-systemd\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:21.779983 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.779880 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-ovnkube-config\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:21.779983 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.779898 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/454a9788-f96e-453d-a83c-63c550c7d96d-agent-certs\") pod \"konnectivity-agent-75hkd\" (UID: \"454a9788-f96e-453d-a83c-63c550c7d96d\") " pod="kube-system/konnectivity-agent-75hkd" Oct 30 21:22:21.779983 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.779926 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6a9841f4-72b0-43fd-8aa7-01393d9a01d1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jtrcb\" (UID: \"6a9841f4-72b0-43fd-8aa7-01393d9a01d1\") " pod="openshift-multus/multus-additional-cni-plugins-jtrcb" Oct 30 21:22:21.779983 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.779947 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/aed61d9f-eb67-44ae-a094-a12814e42559-etc-selinux\") pod \"aws-ebs-csi-driver-node-lvfbv\" (UID: \"aed61d9f-eb67-44ae-a094-a12814e42559\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvfbv" Oct 30 21:22:21.780172 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.779998 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-etc-openvswitch\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:21.780172 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.780021 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a9841f4-72b0-43fd-8aa7-01393d9a01d1-system-cni-dir\") pod \"multus-additional-cni-plugins-jtrcb\" (UID: \"6a9841f4-72b0-43fd-8aa7-01393d9a01d1\") " pod="openshift-multus/multus-additional-cni-plugins-jtrcb" Oct 30 21:22:21.780172 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.780063 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6a9841f4-72b0-43fd-8aa7-01393d9a01d1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jtrcb\" (UID: \"6a9841f4-72b0-43fd-8aa7-01393d9a01d1\") " pod="openshift-multus/multus-additional-cni-plugins-jtrcb" Oct 30 21:22:21.780172 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.780080 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/aed61d9f-eb67-44ae-a094-a12814e42559-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lvfbv\" (UID: \"aed61d9f-eb67-44ae-a094-a12814e42559\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvfbv" Oct 30 21:22:21.780172 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.780129 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-run-openvswitch\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:21.780399 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.780174 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-node-log\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:21.780399 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.780212 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-host-cni-bin\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:21.780399 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.780292 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-env-overrides\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:21.780399 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.780323 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/454a9788-f96e-453d-a83c-63c550c7d96d-konnectivity-ca\") pod \"konnectivity-agent-75hkd\" (UID: \"454a9788-f96e-453d-a83c-63c550c7d96d\") " pod="kube-system/konnectivity-agent-75hkd" Oct 30 21:22:21.780399 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.780357 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6mc5\" (UniqueName: \"kubernetes.io/projected/6a9841f4-72b0-43fd-8aa7-01393d9a01d1-kube-api-access-g6mc5\") pod \"multus-additional-cni-plugins-jtrcb\" (UID: \"6a9841f4-72b0-43fd-8aa7-01393d9a01d1\") " pod="openshift-multus/multus-additional-cni-plugins-jtrcb" Oct 30 21:22:21.780399 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.780397 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83a93696-9f3b-472d-9fc8-92350746df75-metrics-certs\") pod \"network-metrics-daemon-lz8cq\" (UID: \"83a93696-9f3b-472d-9fc8-92350746df75\") " pod="openshift-multus/network-metrics-daemon-lz8cq" Oct 30 21:22:21.780612 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.780456 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-host-run-netns\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:21.780612 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.780534 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/aed61d9f-eb67-44ae-a094-a12814e42559-device-dir\") pod \"aws-ebs-csi-driver-node-lvfbv\" (UID: \"aed61d9f-eb67-44ae-a094-a12814e42559\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvfbv" Oct 30 21:22:21.780612 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.780563 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpbmf\" (UniqueName: \"kubernetes.io/projected/2496f073-7bf0-48af-b091-be6b964117ca-kube-api-access-vpbmf\") pod \"network-check-target-wv464\" (UID: \"2496f073-7bf0-48af-b091-be6b964117ca\") " pod="openshift-network-diagnostics/network-check-target-wv464" Oct 30 21:22:21.780612 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.780580 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-log-socket\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:21.780785 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.780609 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r5f6\" (UniqueName: \"kubernetes.io/projected/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-kube-api-access-6r5f6\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:21.780785 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.780651 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6a9841f4-72b0-43fd-8aa7-01393d9a01d1-cnibin\") pod \"multus-additional-cni-plugins-jtrcb\" (UID: \"6a9841f4-72b0-43fd-8aa7-01393d9a01d1\") " pod="openshift-multus/multus-additional-cni-plugins-jtrcb" Oct 30 21:22:21.780785 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.780692 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6a9841f4-72b0-43fd-8aa7-01393d9a01d1-cni-binary-copy\") pod \"multus-additional-cni-plugins-jtrcb\" (UID: \"6a9841f4-72b0-43fd-8aa7-01393d9a01d1\") " pod="openshift-multus/multus-additional-cni-plugins-jtrcb" Oct 30 21:22:21.780785 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.780720 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8gxg\" (UniqueName: \"kubernetes.io/projected/83a93696-9f3b-472d-9fc8-92350746df75-kube-api-access-b8gxg\") pod \"network-metrics-daemon-lz8cq\" (UID: \"83a93696-9f3b-472d-9fc8-92350746df75\") " pod="openshift-multus/network-metrics-daemon-lz8cq" Oct 30 21:22:21.780785 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.780741 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-var-lib-openvswitch\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:21.780785 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.780756 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-ovn-node-metrics-cert\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:21.781086 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.780798 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6a9841f4-72b0-43fd-8aa7-01393d9a01d1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jtrcb\" (UID: \"6a9841f4-72b0-43fd-8aa7-01393d9a01d1\") " pod="openshift-multus/multus-additional-cni-plugins-jtrcb" Oct 30 21:22:21.781086 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.780845 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/aed61d9f-eb67-44ae-a094-a12814e42559-registration-dir\") pod \"aws-ebs-csi-driver-node-lvfbv\" (UID: \"aed61d9f-eb67-44ae-a094-a12814e42559\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvfbv" Oct 30 21:22:21.781086 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.780873 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4dgx\" (UniqueName: \"kubernetes.io/projected/aed61d9f-eb67-44ae-a094-a12814e42559-kube-api-access-c4dgx\") pod \"aws-ebs-csi-driver-node-lvfbv\" (UID: \"aed61d9f-eb67-44ae-a094-a12814e42559\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvfbv" Oct 30 21:22:21.781086 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.780906 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-systemd-units\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:21.781086 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.780934 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-host-cni-netd\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:21.781086 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.780950 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:21.781086 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.780971 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-ovnkube-script-lib\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:21.781086 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.780996 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/aed61d9f-eb67-44ae-a094-a12814e42559-socket-dir\") pod \"aws-ebs-csi-driver-node-lvfbv\" (UID: \"aed61d9f-eb67-44ae-a094-a12814e42559\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvfbv" Oct 30 21:22:21.781086 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.781025 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/aed61d9f-eb67-44ae-a094-a12814e42559-sys-fs\") pod \"aws-ebs-csi-driver-node-lvfbv\" (UID: \"aed61d9f-eb67-44ae-a094-a12814e42559\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvfbv" Oct 30 21:22:21.781439 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.781404 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-host-kubelet\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:21.781495 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.781448 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-host-slash\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:21.781550 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.781522 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-run-ovn\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:21.781624 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.781599 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-host-run-ovn-kubernetes\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:21.781769 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.781753 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nncz" Oct 30 21:22:21.781860 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:21.781827 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nncz" podUID="6c9c626a-52a7-466e-b744-f4bd79112fd9" Oct 30 21:22:21.784690 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.784667 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n2sfk" Oct 30 21:22:21.795529 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.795488 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-61.us-west-1.compute.internal" event={"ID":"f05411f241c17e725b228582bf16173f","Type":"ContainerStarted","Data":"be8bf0429e5f966f001d3467b942c1f2c894b8e25c67bdba0df0bce3b28e9b59"} Oct 30 21:22:21.796694 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.796667 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-61.us-west-1.compute.internal" event={"ID":"ddfcdf71a39fda8bf039a6129472bc5e","Type":"ContainerStarted","Data":"2e8ccef945c9b2ab26da19f76a836a73756872b7554e0a67985accacdcb37fe9"} Oct 30 21:22:21.813444 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.813423 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 21:22:21.813542 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.813517 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xqzlq\"" Oct 30 21:22:21.813607 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.813582 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 21:22:21.813607 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.813586 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 21:22:21.815752 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.815730 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 21:22:21.815847 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.815748 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 21:22:21.815847 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.815770 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 21:22:21.815847 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.815789 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-gmgqb\"" Oct 30 21:22:21.816224 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.816205 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 21:22:21.816299 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.816223 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 21:22:21.816434 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.816340 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 21:22:21.819388 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.819371 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-ntkmx\"" Oct 30 21:22:21.865786 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.865753 2582 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 21:22:21.865950 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.865809 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-pbd7w" Oct 30 21:22:21.868565 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.868545 2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 21:22:21.882233 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.882203 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-host-slash\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:21.882296 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.882235 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-run-ovn\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:21.882296 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.882259 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d9f7709e-2664-452f-b70b-f4649675781e-hosts-file\") pod \"node-resolver-8rbsm\" (UID: \"d9f7709e-2664-452f-b70b-f4649675781e\") " pod="openshift-dns/node-resolver-8rbsm" Oct 30 21:22:21.882296 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.882282 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptklg\" (UniqueName: \"kubernetes.io/projected/d9f7709e-2664-452f-b70b-f4649675781e-kube-api-access-ptklg\") pod \"node-resolver-8rbsm\" (UID: \"d9f7709e-2664-452f-b70b-f4649675781e\") " pod="openshift-dns/node-resolver-8rbsm" Oct 30 21:22:21.882420 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.882302 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6a9841f4-72b0-43fd-8aa7-01393d9a01d1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jtrcb\" (UID: \"6a9841f4-72b0-43fd-8aa7-01393d9a01d1\") " pod="openshift-multus/multus-additional-cni-plugins-jtrcb" Oct 30 21:22:21.882420 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.882304 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-host-slash\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:21.882420 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.882323 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-run-ovn\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:21.882420 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.882373 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6c9c626a-52a7-466e-b744-f4bd79112fd9-dbus\") pod \"global-pull-secret-syncer-6nncz\" (UID: \"6c9c626a-52a7-466e-b744-f4bd79112fd9\") " pod="kube-system/global-pull-secret-syncer-6nncz" Oct 30 21:22:21.882420 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.882401 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-host-run-netns\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:22:21.882420 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.882417 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-host-var-lib-kubelet\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:22:21.882622 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.882452 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-etc-openvswitch\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:21.882622 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.882490 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-cnibin\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:22:21.882622 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.882497 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-etc-openvswitch\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:21.882622 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.882554 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/aed61d9f-eb67-44ae-a094-a12814e42559-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lvfbv\" (UID: \"aed61d9f-eb67-44ae-a094-a12814e42559\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvfbv" Oct 30 21:22:21.882622 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.882578 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-run-openvswitch\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:21.882622 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.882596 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-env-overrides\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:21.882622 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.882613 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/454a9788-f96e-453d-a83c-63c550c7d96d-konnectivity-ca\") pod \"konnectivity-agent-75hkd\" (UID: \"454a9788-f96e-453d-a83c-63c550c7d96d\") " pod="kube-system/konnectivity-agent-75hkd" Oct 30 21:22:21.882938 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.882643 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6mc5\" (UniqueName: \"kubernetes.io/projected/6a9841f4-72b0-43fd-8aa7-01393d9a01d1-kube-api-access-g6mc5\") pod \"multus-additional-cni-plugins-jtrcb\" (UID: \"6a9841f4-72b0-43fd-8aa7-01393d9a01d1\") " pod="openshift-multus/multus-additional-cni-plugins-jtrcb" Oct 30 21:22:21.882938 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.882653 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-run-openvswitch\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:21.882938 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.882672 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83a93696-9f3b-472d-9fc8-92350746df75-metrics-certs\") pod \"network-metrics-daemon-lz8cq\" (UID: \"83a93696-9f3b-472d-9fc8-92350746df75\") " pod="openshift-multus/network-metrics-daemon-lz8cq" Oct 30 21:22:21.882938 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.882705 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/aed61d9f-eb67-44ae-a094-a12814e42559-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lvfbv\" (UID: \"aed61d9f-eb67-44ae-a094-a12814e42559\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvfbv" Oct 30 21:22:21.882938 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.882812 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-etc-systemd\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:22:21.882938 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.882837 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-system-cni-dir\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:22:21.882938 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.882853 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-etc-modprobe-d\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:22:21.882938 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.882871 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-run\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:22:21.882938 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:21.882869 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:22:21.882938 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.882896 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-host-run-multus-certs\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:22:21.882938 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:21.882962 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83a93696-9f3b-472d-9fc8-92350746df75-metrics-certs podName:83a93696-9f3b-472d-9fc8-92350746df75 nodeName:}" failed. No retries permitted until 2026-10-30 21:22:22.382941003 +0000 UTC m=+3.057109872 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/83a93696-9f3b-472d-9fc8-92350746df75-metrics-certs") pod "network-metrics-daemon-lz8cq" (UID: "83a93696-9f3b-472d-9fc8-92350746df75") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:22:21.883351 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.882982 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-log-socket\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:21.883351 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.883020 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r5f6\" (UniqueName: \"kubernetes.io/projected/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-kube-api-access-6r5f6\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:21.883351 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.883042 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-log-socket\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:21.883351 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.883061 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-ovn-node-metrics-cert\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:21.883351 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.883072 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-env-overrides\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:21.883351 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.883089 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9f7709e-2664-452f-b70b-f4649675781e-tmp-dir\") pod \"node-resolver-8rbsm\" (UID: \"d9f7709e-2664-452f-b70b-f4649675781e\") " pod="openshift-dns/node-resolver-8rbsm" Oct 30 21:22:21.883351 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.883115 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-hostroot\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:22:21.883351 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.883165 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4dgx\" (UniqueName: \"kubernetes.io/projected/aed61d9f-eb67-44ae-a094-a12814e42559-kube-api-access-c4dgx\") pod \"aws-ebs-csi-driver-node-lvfbv\" (UID: \"aed61d9f-eb67-44ae-a094-a12814e42559\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvfbv" Oct 30 21:22:21.883351 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.883198 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-systemd-units\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:21.883351 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.883238 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c3735e3-95ac-4293-8555-38ee9ead0353-host-slash\") pod \"iptables-alerter-mxfq6\" (UID: \"5c3735e3-95ac-4293-8555-38ee9ead0353\") " pod="openshift-network-operator/iptables-alerter-mxfq6" Oct 30 21:22:21.883351 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.883260 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-systemd-units\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:21.883351 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.883284 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-sys\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:22:21.883351 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.883321 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/aed61d9f-eb67-44ae-a094-a12814e42559-sys-fs\") pod \"aws-ebs-csi-driver-node-lvfbv\" (UID: \"aed61d9f-eb67-44ae-a094-a12814e42559\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvfbv" Oct 30 21:22:21.883351 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.883343 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-host-run-ovn-kubernetes\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:21.884048 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.883384 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-host-run-ovn-kubernetes\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:21.884048 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.883397 2582 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 21:22:21.884048 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.883401 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-lib-modules\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:22:21.884048 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.883420 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/aed61d9f-eb67-44ae-a094-a12814e42559-sys-fs\") pod \"aws-ebs-csi-driver-node-lvfbv\" (UID: \"aed61d9f-eb67-44ae-a094-a12814e42559\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvfbv" Oct 30 21:22:21.884048 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.883453 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-run-systemd\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:21.884048 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.883486 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-ovnkube-config\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:21.884048 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.883523 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-run-systemd\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:21.884048 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.883538 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/454a9788-f96e-453d-a83c-63c550c7d96d-agent-certs\") pod \"konnectivity-agent-75hkd\" (UID: \"454a9788-f96e-453d-a83c-63c550c7d96d\") " pod="kube-system/konnectivity-agent-75hkd" Oct 30 21:22:21.884048 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.883580 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq8kj\" (UniqueName: \"kubernetes.io/projected/5c3735e3-95ac-4293-8555-38ee9ead0353-kube-api-access-bq8kj\") pod \"iptables-alerter-mxfq6\" (UID: \"5c3735e3-95ac-4293-8555-38ee9ead0353\") " pod="openshift-network-operator/iptables-alerter-mxfq6" Oct 30 21:22:21.884048 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.883613 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-etc-sysconfig\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:22:21.884048 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.883643 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/59571898-f15b-4b8f-b043-0e5af369c39e-serviceca\") pod \"node-ca-8vqf9\" (UID: \"59571898-f15b-4b8f-b043-0e5af369c39e\") " pod="openshift-image-registry/node-ca-8vqf9" Oct 30 21:22:21.884048 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.883668 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3cabcccd-6b1e-4089-a3a6-275558de2bfb-multus-daemon-config\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:22:21.884048 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.883702 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/aed61d9f-eb67-44ae-a094-a12814e42559-etc-selinux\") pod \"aws-ebs-csi-driver-node-lvfbv\" (UID: \"aed61d9f-eb67-44ae-a094-a12814e42559\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvfbv" Oct 30 21:22:21.884048 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.883732 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a9841f4-72b0-43fd-8aa7-01393d9a01d1-system-cni-dir\") pod \"multus-additional-cni-plugins-jtrcb\" (UID: \"6a9841f4-72b0-43fd-8aa7-01393d9a01d1\") " pod="openshift-multus/multus-additional-cni-plugins-jtrcb" Oct 30 21:22:21.884048 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.883766 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6a9841f4-72b0-43fd-8aa7-01393d9a01d1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jtrcb\" (UID: \"6a9841f4-72b0-43fd-8aa7-01393d9a01d1\") " pod="openshift-multus/multus-additional-cni-plugins-jtrcb" Oct 30 21:22:21.884048 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.883800 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-etc-sysctl-conf\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:22:21.884048 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.883810 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a9841f4-72b0-43fd-8aa7-01393d9a01d1-system-cni-dir\") pod \"multus-additional-cni-plugins-jtrcb\" (UID: \"6a9841f4-72b0-43fd-8aa7-01393d9a01d1\") " pod="openshift-multus/multus-additional-cni-plugins-jtrcb" Oct 30 21:22:21.884869 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.883835 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/aed61d9f-eb67-44ae-a094-a12814e42559-etc-selinux\") pod \"aws-ebs-csi-driver-node-lvfbv\" (UID: \"aed61d9f-eb67-44ae-a094-a12814e42559\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvfbv" Oct 30 21:22:21.884869 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.883827 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8ea6188f-9e30-4ab0-b5b3-c99647377e00-etc-tuned\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:22:21.884869 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.883876 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ea6188f-9e30-4ab0-b5b3-c99647377e00-tmp\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:22:21.884869 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.883925 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7qhc\" (UniqueName: \"kubernetes.io/projected/59571898-f15b-4b8f-b043-0e5af369c39e-kube-api-access-d7qhc\") pod \"node-ca-8vqf9\" (UID: \"59571898-f15b-4b8f-b043-0e5af369c39e\") " pod="openshift-image-registry/node-ca-8vqf9" Oct 30 21:22:21.884869 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.883972 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6c9c626a-52a7-466e-b744-f4bd79112fd9-kubelet-config\") pod \"global-pull-secret-syncer-6nncz\" (UID: \"6c9c626a-52a7-466e-b744-f4bd79112fd9\") " pod="kube-system/global-pull-secret-syncer-6nncz" Oct 30 21:22:21.884869 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.884007 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-node-log\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:21.884869 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.884086 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-host-cni-bin\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:21.884869 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.884106 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-node-log\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:21.884869 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.884095 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-ovnkube-config\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:21.884869 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.884122 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-host-cni-bin\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:21.884869 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.884140 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-host\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:22:21.884869 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.884170 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6c9c626a-52a7-466e-b744-f4bd79112fd9-original-pull-secret\") pod \"global-pull-secret-syncer-6nncz\" (UID: \"6c9c626a-52a7-466e-b744-f4bd79112fd9\") " pod="kube-system/global-pull-secret-syncer-6nncz" Oct 30 21:22:21.884869 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.884203 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3cabcccd-6b1e-4089-a3a6-275558de2bfb-cni-binary-copy\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:22:21.884869 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.884229 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-multus-socket-dir-parent\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:22:21.884869 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.884277 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-etc-kubernetes\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:22:21.884869 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.884336 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-host-run-netns\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:21.884869 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.884407 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/aed61d9f-eb67-44ae-a094-a12814e42559-device-dir\") pod \"aws-ebs-csi-driver-node-lvfbv\" (UID: \"aed61d9f-eb67-44ae-a094-a12814e42559\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvfbv" Oct 30 21:22:21.885659 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.884410 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-host-run-netns\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:21.885659 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.884437 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpbmf\" (UniqueName: \"kubernetes.io/projected/2496f073-7bf0-48af-b091-be6b964117ca-kube-api-access-vpbmf\") pod \"network-check-target-wv464\" (UID: \"2496f073-7bf0-48af-b091-be6b964117ca\") " pod="openshift-network-diagnostics/network-check-target-wv464" Oct 30 21:22:21.885659 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.884447 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/aed61d9f-eb67-44ae-a094-a12814e42559-device-dir\") pod \"aws-ebs-csi-driver-node-lvfbv\" (UID: \"aed61d9f-eb67-44ae-a094-a12814e42559\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvfbv" Oct 30 21:22:21.885659 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.884468 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-etc-sysctl-d\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:22:21.885659 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.884530 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/59571898-f15b-4b8f-b043-0e5af369c39e-host\") pod \"node-ca-8vqf9\" (UID: \"59571898-f15b-4b8f-b043-0e5af369c39e\") " pod="openshift-image-registry/node-ca-8vqf9" Oct 30 21:22:21.885659 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.884564 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6a9841f4-72b0-43fd-8aa7-01393d9a01d1-cnibin\") pod \"multus-additional-cni-plugins-jtrcb\" (UID: \"6a9841f4-72b0-43fd-8aa7-01393d9a01d1\") " pod="openshift-multus/multus-additional-cni-plugins-jtrcb" Oct 30 21:22:21.885659 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.884656 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6a9841f4-72b0-43fd-8aa7-01393d9a01d1-cnibin\") pod \"multus-additional-cni-plugins-jtrcb\" (UID: \"6a9841f4-72b0-43fd-8aa7-01393d9a01d1\") " pod="openshift-multus/multus-additional-cni-plugins-jtrcb" Oct 30 21:22:21.885659 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.884689 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6a9841f4-72b0-43fd-8aa7-01393d9a01d1-cni-binary-copy\") pod \"multus-additional-cni-plugins-jtrcb\" (UID: \"6a9841f4-72b0-43fd-8aa7-01393d9a01d1\") " pod="openshift-multus/multus-additional-cni-plugins-jtrcb" Oct 30 21:22:21.885659 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.884743 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8gxg\" (UniqueName: \"kubernetes.io/projected/83a93696-9f3b-472d-9fc8-92350746df75-kube-api-access-b8gxg\") pod \"network-metrics-daemon-lz8cq\" (UID: \"83a93696-9f3b-472d-9fc8-92350746df75\") " pod="openshift-multus/network-metrics-daemon-lz8cq" Oct 30 21:22:21.885659 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.884832 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-host-var-lib-cni-multus\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:22:21.885659 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.884878 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-var-lib-openvswitch\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:21.885659 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.884911 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6a9841f4-72b0-43fd-8aa7-01393d9a01d1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jtrcb\" (UID: \"6a9841f4-72b0-43fd-8aa7-01393d9a01d1\") " pod="openshift-multus/multus-additional-cni-plugins-jtrcb" Oct 30 21:22:21.885659 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.884959 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5c3735e3-95ac-4293-8555-38ee9ead0353-iptables-alerter-script\") pod \"iptables-alerter-mxfq6\" (UID: \"5c3735e3-95ac-4293-8555-38ee9ead0353\") " pod="openshift-network-operator/iptables-alerter-mxfq6" Oct 30 21:22:21.885659 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.884978 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-var-lib-openvswitch\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:21.885659 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.884989 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-host-run-k8s-cni-cncf-io\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:22:21.885659 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.885081 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-multus-conf-dir\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:22:21.885659 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.885128 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blx28\" (UniqueName: \"kubernetes.io/projected/3cabcccd-6b1e-4089-a3a6-275558de2bfb-kube-api-access-blx28\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:22:21.886209 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.885144 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6a9841f4-72b0-43fd-8aa7-01393d9a01d1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jtrcb\" (UID: \"6a9841f4-72b0-43fd-8aa7-01393d9a01d1\") " pod="openshift-multus/multus-additional-cni-plugins-jtrcb" Oct 30 21:22:21.886209 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.885180 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/aed61d9f-eb67-44ae-a094-a12814e42559-registration-dir\") pod \"aws-ebs-csi-driver-node-lvfbv\" (UID: \"aed61d9f-eb67-44ae-a094-a12814e42559\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvfbv" Oct 30 21:22:21.886209 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.885220 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-host-cni-netd\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:21.886209 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.885267 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/aed61d9f-eb67-44ae-a094-a12814e42559-registration-dir\") pod \"aws-ebs-csi-driver-node-lvfbv\" (UID: \"aed61d9f-eb67-44ae-a094-a12814e42559\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvfbv" Oct 30 21:22:21.886209 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.885266 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:21.886209 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.885322 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-ovnkube-script-lib\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:21.886209 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.885330 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-host-cni-netd\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:21.886209 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.885351 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-etc-kubernetes\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:22:21.886209 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.885347 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:21.886209 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.885382 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4v76\" (UniqueName: \"kubernetes.io/projected/8ea6188f-9e30-4ab0-b5b3-c99647377e00-kube-api-access-b4v76\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:22:21.886209 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.885406 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-multus-cni-dir\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:22:21.886209 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.885428 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-host-var-lib-cni-bin\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:22:21.886209 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.885467 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-var-lib-kubelet\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:22:21.886209 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.885508 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/aed61d9f-eb67-44ae-a094-a12814e42559-socket-dir\") pod \"aws-ebs-csi-driver-node-lvfbv\" (UID: \"aed61d9f-eb67-44ae-a094-a12814e42559\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvfbv" Oct 30 21:22:21.886209 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.885592 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-host-kubelet\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:21.886209 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.885597 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/aed61d9f-eb67-44ae-a094-a12814e42559-socket-dir\") pod \"aws-ebs-csi-driver-node-lvfbv\" (UID: \"aed61d9f-eb67-44ae-a094-a12814e42559\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvfbv" Oct 30 21:22:21.886209 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.885670 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-host-kubelet\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:21.886684 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.885772 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-ovnkube-script-lib\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:21.887486 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.887470 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-ovn-node-metrics-cert\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:21.922968 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.922876 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 21:22:21.922968 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.922918 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 21:22:21.922968 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.922933 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 21:22:21.923548 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.923280 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 21:22:21.923548 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.923308 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-gkfxf\"" Oct 30 21:22:21.923548 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.923306 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:22:21.923548 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.923311 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:22:21.923548 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.923398 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 21:22:21.923548 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.923403 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 21:22:21.923548 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.923433 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 21:22:21.923548 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.923521 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 21:22:21.924046 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.923578 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 21:22:21.924046 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.923615 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 21:22:21.924046 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.923725 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 21:22:21.924046 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.923758 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 21:22:21.924046 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.923828 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 21:22:21.924046 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.923847 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7lclr\"" Oct 30 21:22:21.924046 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.923867 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-v6bcb\"" Oct 30 21:22:21.924046 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.923869 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 21:22:21.924046 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.923886 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-vm9q9\"" Oct 30 21:22:21.924046 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.923930 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-rf4nn\"" Oct 30 21:22:21.924374 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.924057 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 21:22:21.924374 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.924137 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-mbjr2\"" Oct 30 21:22:21.924374 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.924169 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 21:22:21.924476 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.924458 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6a9841f4-72b0-43fd-8aa7-01393d9a01d1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jtrcb\" (UID: \"6a9841f4-72b0-43fd-8aa7-01393d9a01d1\") " pod="openshift-multus/multus-additional-cni-plugins-jtrcb" Oct 30 21:22:21.926170 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.926153 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6a9841f4-72b0-43fd-8aa7-01393d9a01d1-cni-binary-copy\") pod \"multus-additional-cni-plugins-jtrcb\" (UID: \"6a9841f4-72b0-43fd-8aa7-01393d9a01d1\") " pod="openshift-multus/multus-additional-cni-plugins-jtrcb" Oct 30 21:22:21.932816 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.932797 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6a9841f4-72b0-43fd-8aa7-01393d9a01d1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jtrcb\" (UID: \"6a9841f4-72b0-43fd-8aa7-01393d9a01d1\") " pod="openshift-multus/multus-additional-cni-plugins-jtrcb" Oct 30 21:22:21.933221 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.933197 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/454a9788-f96e-453d-a83c-63c550c7d96d-konnectivity-ca\") pod \"konnectivity-agent-75hkd\" (UID: \"454a9788-f96e-453d-a83c-63c550c7d96d\") " pod="kube-system/konnectivity-agent-75hkd" Oct 30 21:22:21.935878 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.935861 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/454a9788-f96e-453d-a83c-63c550c7d96d-agent-certs\") pod \"konnectivity-agent-75hkd\" (UID: \"454a9788-f96e-453d-a83c-63c550c7d96d\") " pod="kube-system/konnectivity-agent-75hkd" Oct 30 21:22:21.948230 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:21.948206 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:22:21.948230 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:21.948234 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:22:21.948398 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:21.948247 2582 projected.go:196] Error preparing data for projected volume kube-api-access-vpbmf for pod openshift-network-diagnostics/network-check-target-wv464: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:22:21.948398 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:21.948321 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2496f073-7bf0-48af-b091-be6b964117ca-kube-api-access-vpbmf podName:2496f073-7bf0-48af-b091-be6b964117ca nodeName:}" failed. No retries permitted until 2026-10-30 21:22:22.448301656 +0000 UTC m=+3.122470541 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-vpbmf" (UniqueName: "kubernetes.io/projected/2496f073-7bf0-48af-b091-be6b964117ca-kube-api-access-vpbmf") pod "network-check-target-wv464" (UID: "2496f073-7bf0-48af-b091-be6b964117ca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:22:21.950476 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.950448 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6mc5\" (UniqueName: \"kubernetes.io/projected/6a9841f4-72b0-43fd-8aa7-01393d9a01d1-kube-api-access-g6mc5\") pod \"multus-additional-cni-plugins-jtrcb\" (UID: \"6a9841f4-72b0-43fd-8aa7-01393d9a01d1\") " pod="openshift-multus/multus-additional-cni-plugins-jtrcb" Oct 30 21:22:21.950582 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.950456 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8gxg\" (UniqueName: \"kubernetes.io/projected/83a93696-9f3b-472d-9fc8-92350746df75-kube-api-access-b8gxg\") pod \"network-metrics-daemon-lz8cq\" (UID: \"83a93696-9f3b-472d-9fc8-92350746df75\") " pod="openshift-multus/network-metrics-daemon-lz8cq" Oct 30 21:22:21.950582 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.950565 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4dgx\" (UniqueName: \"kubernetes.io/projected/aed61d9f-eb67-44ae-a094-a12814e42559-kube-api-access-c4dgx\") pod \"aws-ebs-csi-driver-node-lvfbv\" (UID: \"aed61d9f-eb67-44ae-a094-a12814e42559\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvfbv" Oct 30 21:22:21.950664 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.950614 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r5f6\" (UniqueName: \"kubernetes.io/projected/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-kube-api-access-6r5f6\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:21.986953 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.986896 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-etc-kubernetes\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:22:21.986953 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.986961 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4v76\" (UniqueName: \"kubernetes.io/projected/8ea6188f-9e30-4ab0-b5b3-c99647377e00-kube-api-access-b4v76\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:22:21.987210 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.986985 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-multus-cni-dir\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:22:21.987210 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.987023 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-etc-kubernetes\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:22:21.987210 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.987010 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-host-var-lib-cni-bin\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:22:21.987210 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.987099 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-var-lib-kubelet\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:22:21.987210 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.987121 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-host-var-lib-cni-bin\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:22:21.987210 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.987144 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d9f7709e-2664-452f-b70b-f4649675781e-hosts-file\") pod \"node-resolver-8rbsm\" (UID: \"d9f7709e-2664-452f-b70b-f4649675781e\") " pod="openshift-dns/node-resolver-8rbsm" Oct 30 21:22:21.987210 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.987158 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-var-lib-kubelet\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:22:21.987210 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.987192 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-multus-cni-dir\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:22:21.987210 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.987197 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptklg\" (UniqueName: \"kubernetes.io/projected/d9f7709e-2664-452f-b70b-f4649675781e-kube-api-access-ptklg\") pod \"node-resolver-8rbsm\" (UID: \"d9f7709e-2664-452f-b70b-f4649675781e\") " pod="openshift-dns/node-resolver-8rbsm" Oct 30 21:22:21.987629 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.987245 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6c9c626a-52a7-466e-b744-f4bd79112fd9-dbus\") pod \"global-pull-secret-syncer-6nncz\" (UID: \"6c9c626a-52a7-466e-b744-f4bd79112fd9\") " pod="kube-system/global-pull-secret-syncer-6nncz" Oct 30 21:22:21.987629 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.987246 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d9f7709e-2664-452f-b70b-f4649675781e-hosts-file\") pod \"node-resolver-8rbsm\" (UID: \"d9f7709e-2664-452f-b70b-f4649675781e\") " pod="openshift-dns/node-resolver-8rbsm" Oct 30 21:22:21.987629 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.987277 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-host-run-netns\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:22:21.987629 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.987298 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-host-var-lib-kubelet\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:22:21.987629 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.987318 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-cnibin\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:22:21.987629 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.987348 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6c9c626a-52a7-466e-b744-f4bd79112fd9-dbus\") pod \"global-pull-secret-syncer-6nncz\" (UID: \"6c9c626a-52a7-466e-b744-f4bd79112fd9\") " pod="kube-system/global-pull-secret-syncer-6nncz" Oct 30 21:22:21.987629 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.987319 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-host-run-netns\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:22:21.987629 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.987368 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-etc-systemd\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:22:21.987629 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.987389 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-system-cni-dir\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:22:21.987629 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.987408 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-etc-modprobe-d\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:22:21.987629 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.987412 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-host-var-lib-kubelet\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:22:21.987629 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.987423 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-run\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:22:21.987629 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.987433 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-cnibin\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:22:21.987629 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.987447 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-host-run-multus-certs\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:22:21.987629 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.987479 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-system-cni-dir\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:22:21.987629 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.987488 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-etc-modprobe-d\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:22:21.987629 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.987503 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-run\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:22:21.987629 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.987505 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-host-run-multus-certs\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:22:21.988550 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.987526 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9f7709e-2664-452f-b70b-f4649675781e-tmp-dir\") pod \"node-resolver-8rbsm\" (UID: \"d9f7709e-2664-452f-b70b-f4649675781e\") " pod="openshift-dns/node-resolver-8rbsm" Oct 30 21:22:21.988550 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.987537 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-etc-systemd\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:22:21.988550 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.987545 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-hostroot\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:22:21.988550 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.987571 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-hostroot\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:22:21.988550 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.987586 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c3735e3-95ac-4293-8555-38ee9ead0353-host-slash\") pod \"iptables-alerter-mxfq6\" (UID: \"5c3735e3-95ac-4293-8555-38ee9ead0353\") " pod="openshift-network-operator/iptables-alerter-mxfq6" Oct 30 21:22:21.988550 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.987602 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-sys\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:22:21.988550 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.987646 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-lib-modules\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:22:21.988550 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.987659 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c3735e3-95ac-4293-8555-38ee9ead0353-host-slash\") pod \"iptables-alerter-mxfq6\" (UID: \"5c3735e3-95ac-4293-8555-38ee9ead0353\") " pod="openshift-network-operator/iptables-alerter-mxfq6" Oct 30 21:22:21.988550 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.987666 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq8kj\" (UniqueName: \"kubernetes.io/projected/5c3735e3-95ac-4293-8555-38ee9ead0353-kube-api-access-bq8kj\") pod \"iptables-alerter-mxfq6\" (UID: \"5c3735e3-95ac-4293-8555-38ee9ead0353\") " pod="openshift-network-operator/iptables-alerter-mxfq6" Oct 30 21:22:21.988550 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.987685 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-etc-sysconfig\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:22:21.988550 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.987715 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-sys\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:22:21.988550 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.987760 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-etc-sysconfig\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:22:21.988550 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.987784 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-lib-modules\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:22:21.988550 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.987815 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/59571898-f15b-4b8f-b043-0e5af369c39e-serviceca\") pod \"node-ca-8vqf9\" (UID: \"59571898-f15b-4b8f-b043-0e5af369c39e\") " pod="openshift-image-registry/node-ca-8vqf9" Oct 30 21:22:21.988550 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.987828 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9f7709e-2664-452f-b70b-f4649675781e-tmp-dir\") pod \"node-resolver-8rbsm\" (UID: \"d9f7709e-2664-452f-b70b-f4649675781e\") " pod="openshift-dns/node-resolver-8rbsm" Oct 30 21:22:21.988550 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.987843 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3cabcccd-6b1e-4089-a3a6-275558de2bfb-multus-daemon-config\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:22:21.988550 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.987885 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-etc-sysctl-conf\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:22:21.988550 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.987932 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8ea6188f-9e30-4ab0-b5b3-c99647377e00-etc-tuned\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:22:21.989381 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.987955 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ea6188f-9e30-4ab0-b5b3-c99647377e00-tmp\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:22:21.989381 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.987975 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7qhc\" (UniqueName: \"kubernetes.io/projected/59571898-f15b-4b8f-b043-0e5af369c39e-kube-api-access-d7qhc\") pod \"node-ca-8vqf9\" (UID: \"59571898-f15b-4b8f-b043-0e5af369c39e\") " pod="openshift-image-registry/node-ca-8vqf9" Oct 30 21:22:21.989381 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.987997 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6c9c626a-52a7-466e-b744-f4bd79112fd9-kubelet-config\") pod \"global-pull-secret-syncer-6nncz\" (UID: \"6c9c626a-52a7-466e-b744-f4bd79112fd9\") " pod="kube-system/global-pull-secret-syncer-6nncz" Oct 30 21:22:21.989381 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.988019 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-host\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:22:21.989381 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.988050 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6c9c626a-52a7-466e-b744-f4bd79112fd9-original-pull-secret\") pod \"global-pull-secret-syncer-6nncz\" (UID: \"6c9c626a-52a7-466e-b744-f4bd79112fd9\") " pod="kube-system/global-pull-secret-syncer-6nncz" Oct 30 21:22:21.989381 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.988065 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6c9c626a-52a7-466e-b744-f4bd79112fd9-kubelet-config\") pod \"global-pull-secret-syncer-6nncz\" (UID: \"6c9c626a-52a7-466e-b744-f4bd79112fd9\") " pod="kube-system/global-pull-secret-syncer-6nncz" Oct 30 21:22:21.989381 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.988081 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3cabcccd-6b1e-4089-a3a6-275558de2bfb-cni-binary-copy\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:22:21.989381 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.988018 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-etc-sysctl-conf\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:22:21.989381 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.988112 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-multus-socket-dir-parent\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:22:21.989381 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.988136 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-host\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:22:21.989381 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.988139 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-etc-kubernetes\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:22:21.989381 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.988187 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-etc-sysctl-d\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:22:21.989381 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.988215 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/59571898-f15b-4b8f-b043-0e5af369c39e-host\") pod \"node-ca-8vqf9\" (UID: \"59571898-f15b-4b8f-b043-0e5af369c39e\") " pod="openshift-image-registry/node-ca-8vqf9" Oct 30 21:22:21.989381 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.988265 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-host-var-lib-cni-multus\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:22:21.989381 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.988295 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5c3735e3-95ac-4293-8555-38ee9ead0353-iptables-alerter-script\") pod \"iptables-alerter-mxfq6\" (UID: \"5c3735e3-95ac-4293-8555-38ee9ead0353\") " pod="openshift-network-operator/iptables-alerter-mxfq6" Oct 30 21:22:21.989381 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.988321 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-host-run-k8s-cni-cncf-io\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:22:21.989381 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.988350 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-multus-conf-dir\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:22:21.989381 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.988381 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blx28\" (UniqueName: \"kubernetes.io/projected/3cabcccd-6b1e-4089-a3a6-275558de2bfb-kube-api-access-blx28\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:22:21.990248 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.988392 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/59571898-f15b-4b8f-b043-0e5af369c39e-serviceca\") pod \"node-ca-8vqf9\" (UID: \"59571898-f15b-4b8f-b043-0e5af369c39e\") " pod="openshift-image-registry/node-ca-8vqf9" Oct 30 21:22:21.990248 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.988489 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-host-var-lib-cni-multus\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:22:21.990248 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.988492 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-etc-sysctl-d\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:22:21.990248 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.988528 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3cabcccd-6b1e-4089-a3a6-275558de2bfb-multus-daemon-config\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:22:21.990248 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.989098 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3cabcccd-6b1e-4089-a3a6-275558de2bfb-cni-binary-copy\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:22:21.990248 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.989101 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-multus-socket-dir-parent\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:22:21.990248 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.989219 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/59571898-f15b-4b8f-b043-0e5af369c39e-host\") pod \"node-ca-8vqf9\" (UID: \"59571898-f15b-4b8f-b043-0e5af369c39e\") " pod="openshift-image-registry/node-ca-8vqf9" Oct 30 21:22:21.990248 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.989277 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-host-run-k8s-cni-cncf-io\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:22:21.990248 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.989328 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-multus-conf-dir\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:22:21.990248 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.989358 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-etc-kubernetes\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:22:21.990248 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:21.989511 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:22:21.990248 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:21.989726 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c9c626a-52a7-466e-b744-f4bd79112fd9-original-pull-secret podName:6c9c626a-52a7-466e-b744-f4bd79112fd9 nodeName:}" failed. No retries permitted until 2026-10-30 21:22:22.489702964 +0000 UTC m=+3.163871848 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6c9c626a-52a7-466e-b744-f4bd79112fd9-original-pull-secret") pod "global-pull-secret-syncer-6nncz" (UID: "6c9c626a-52a7-466e-b744-f4bd79112fd9") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:22:21.990248 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.990030 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5c3735e3-95ac-4293-8555-38ee9ead0353-iptables-alerter-script\") pod \"iptables-alerter-mxfq6\" (UID: \"5c3735e3-95ac-4293-8555-38ee9ead0353\") " pod="openshift-network-operator/iptables-alerter-mxfq6" Oct 30 21:22:21.992436 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.991392 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8ea6188f-9e30-4ab0-b5b3-c99647377e00-etc-tuned\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:22:21.992436 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:21.991439 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ea6188f-9e30-4ab0-b5b3-c99647377e00-tmp\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:22:22.054023 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:22.053982 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq8kj\" (UniqueName: \"kubernetes.io/projected/5c3735e3-95ac-4293-8555-38ee9ead0353-kube-api-access-bq8kj\") pod \"iptables-alerter-mxfq6\" (UID: \"5c3735e3-95ac-4293-8555-38ee9ead0353\") " pod="openshift-network-operator/iptables-alerter-mxfq6" Oct 30 21:22:22.054204 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:22.054118 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptklg\" (UniqueName: \"kubernetes.io/projected/d9f7709e-2664-452f-b70b-f4649675781e-kube-api-access-ptklg\") pod \"node-resolver-8rbsm\" (UID: \"d9f7709e-2664-452f-b70b-f4649675781e\") " pod="openshift-dns/node-resolver-8rbsm" Oct 30 21:22:22.054204 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:22.054160 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4v76\" (UniqueName: \"kubernetes.io/projected/8ea6188f-9e30-4ab0-b5b3-c99647377e00-kube-api-access-b4v76\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:22:22.054356 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:22.054248 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7qhc\" (UniqueName: \"kubernetes.io/projected/59571898-f15b-4b8f-b043-0e5af369c39e-kube-api-access-d7qhc\") pod \"node-ca-8vqf9\" (UID: \"59571898-f15b-4b8f-b043-0e5af369c39e\") " pod="openshift-image-registry/node-ca-8vqf9" Oct 30 21:22:22.054447 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:22.054426 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blx28\" (UniqueName: \"kubernetes.io/projected/3cabcccd-6b1e-4089-a3a6-275558de2bfb-kube-api-access-blx28\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:22:22.073441 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:22.073406 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvfbv" Oct 30 21:22:22.084445 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:22.084411 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaed61d9f_eb67_44ae_a094_a12814e42559.slice/crio-253902cf7fb7a07a5987fe209537eb45f982a10286b6a15f35eacb7027da2cd8 WatchSource:0}: Error finding container 253902cf7fb7a07a5987fe209537eb45f982a10286b6a15f35eacb7027da2cd8: Status 404 returned error can't find the container with id 253902cf7fb7a07a5987fe209537eb45f982a10286b6a15f35eacb7027da2cd8 Oct 30 21:22:22.090318 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:22.090298 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-75hkd" Oct 30 21:22:22.100343 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:22.100312 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod454a9788_f96e_453d_a83c_63c550c7d96d.slice/crio-6b2a31073c7737830fb55f6fe12435407df6541f64cf11219ac235d3ab7db600 WatchSource:0}: Error finding container 6b2a31073c7737830fb55f6fe12435407df6541f64cf11219ac235d3ab7db600: Status 404 returned error can't find the container with id 6b2a31073c7737830fb55f6fe12435407df6541f64cf11219ac235d3ab7db600 Oct 30 21:22:22.103828 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:22.103808 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mxfq6" Oct 30 21:22:22.104939 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:22.104920 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:22.108659 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:22.108641 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:22:22.111152 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:22.111125 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8rbsm" Oct 30 21:22:22.113432 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:22.113401 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c3735e3_95ac_4293_8555_38ee9ead0353.slice/crio-e1b78fb648faa00672c441f6ea61fae944f1bcd635af480835decf0a41cc8f60 WatchSource:0}: Error finding container e1b78fb648faa00672c441f6ea61fae944f1bcd635af480835decf0a41cc8f60: Status 404 returned error can't find the container with id e1b78fb648faa00672c441f6ea61fae944f1bcd635af480835decf0a41cc8f60 Oct 30 21:22:22.118604 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:22.118578 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3df036ff_1c00_4ce2_9e86_8acbeedf6c82.slice/crio-9dc54f963eddd9f65ef3f9797b5563e449fed9af9fc472616366c4b6d508a75e WatchSource:0}: Error finding container 9dc54f963eddd9f65ef3f9797b5563e449fed9af9fc472616366c4b6d508a75e: Status 404 returned error can't find the container with id 9dc54f963eddd9f65ef3f9797b5563e449fed9af9fc472616366c4b6d508a75e Oct 30 21:22:22.119722 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:22.119704 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8vqf9" Oct 30 21:22:22.121731 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:22.121708 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ea6188f_9e30_4ab0_b5b3_c99647377e00.slice/crio-53209f7dc29327037916c256cb46bf6239e5ddba43ea43c634c6eb30c579884f WatchSource:0}: Error finding container 53209f7dc29327037916c256cb46bf6239e5ddba43ea43c634c6eb30c579884f: Status 404 returned error can't find the container with id 53209f7dc29327037916c256cb46bf6239e5ddba43ea43c634c6eb30c579884f Oct 30 21:22:22.122114 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:22.122094 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n2sfk" Oct 30 21:22:22.124101 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:22.124087 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jtrcb" Oct 30 21:22:22.125374 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:22.125351 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9f7709e_2664_452f_b70b_f4649675781e.slice/crio-21dac7bbe759d29d25155018511b5532b1b78d6e336d26183d4d7505e28570ab WatchSource:0}: Error finding container 21dac7bbe759d29d25155018511b5532b1b78d6e336d26183d4d7505e28570ab: Status 404 returned error can't find the container with id 21dac7bbe759d29d25155018511b5532b1b78d6e336d26183d4d7505e28570ab Oct 30 21:22:22.130663 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:22.130642 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59571898_f15b_4b8f_b043_0e5af369c39e.slice/crio-b092a4deb6fd47f6577074ba533fcd9831761318517be661b91db0c2191c919f WatchSource:0}: Error finding container b092a4deb6fd47f6577074ba533fcd9831761318517be661b91db0c2191c919f: Status 404 returned error can't find the container with id b092a4deb6fd47f6577074ba533fcd9831761318517be661b91db0c2191c919f Oct 30 21:22:22.135582 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:22.135554 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cabcccd_6b1e_4089_a3a6_275558de2bfb.slice/crio-ed963bfa9bbb231b5afbfba50c8b3914a24934b413908bda4c6f037527000a01 WatchSource:0}: Error finding container ed963bfa9bbb231b5afbfba50c8b3914a24934b413908bda4c6f037527000a01: Status 404 returned error can't find the container with id ed963bfa9bbb231b5afbfba50c8b3914a24934b413908bda4c6f037527000a01 Oct 30 21:22:22.136996 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:22:22.136905 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a9841f4_72b0_43fd_8aa7_01393d9a01d1.slice/crio-953276e6631de9919ab79f85e9cad674e90cb10dc00b94a70fea3bf5cef97545 WatchSource:0}: Error finding container 953276e6631de9919ab79f85e9cad674e90cb10dc00b94a70fea3bf5cef97545: Status 404 returned error can't find the container with id 953276e6631de9919ab79f85e9cad674e90cb10dc00b94a70fea3bf5cef97545 Oct 30 21:22:22.170361 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:22.170325 2582 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:22:22.391966 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:22.391908 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83a93696-9f3b-472d-9fc8-92350746df75-metrics-certs\") pod \"network-metrics-daemon-lz8cq\" (UID: \"83a93696-9f3b-472d-9fc8-92350746df75\") " pod="openshift-multus/network-metrics-daemon-lz8cq" Oct 30 21:22:22.392150 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:22.392111 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:22:22.392193 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:22.392182 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83a93696-9f3b-472d-9fc8-92350746df75-metrics-certs podName:83a93696-9f3b-472d-9fc8-92350746df75 nodeName:}" failed. No retries permitted until 2026-10-30 21:22:23.392165059 +0000 UTC m=+4.066333940 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/83a93696-9f3b-472d-9fc8-92350746df75-metrics-certs") pod "network-metrics-daemon-lz8cq" (UID: "83a93696-9f3b-472d-9fc8-92350746df75") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:22:22.493414 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:22.493370 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6c9c626a-52a7-466e-b744-f4bd79112fd9-original-pull-secret\") pod \"global-pull-secret-syncer-6nncz\" (UID: \"6c9c626a-52a7-466e-b744-f4bd79112fd9\") " pod="kube-system/global-pull-secret-syncer-6nncz" Oct 30 21:22:22.493603 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:22.493433 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpbmf\" (UniqueName: \"kubernetes.io/projected/2496f073-7bf0-48af-b091-be6b964117ca-kube-api-access-vpbmf\") pod \"network-check-target-wv464\" (UID: \"2496f073-7bf0-48af-b091-be6b964117ca\") " pod="openshift-network-diagnostics/network-check-target-wv464" Oct 30 21:22:22.493603 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:22.493541 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:22:22.493603 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:22.493586 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:22:22.493603 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:22.493606 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:22:22.493840 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:22.493620 2582 projected.go:196] Error preparing data for projected volume kube-api-access-vpbmf for pod openshift-network-diagnostics/network-check-target-wv464: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:22:22.493840 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:22.493623 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c9c626a-52a7-466e-b744-f4bd79112fd9-original-pull-secret podName:6c9c626a-52a7-466e-b744-f4bd79112fd9 nodeName:}" failed. No retries permitted until 2026-10-30 21:22:23.493602179 +0000 UTC m=+4.167771048 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6c9c626a-52a7-466e-b744-f4bd79112fd9-original-pull-secret") pod "global-pull-secret-syncer-6nncz" (UID: "6c9c626a-52a7-466e-b744-f4bd79112fd9") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:22:22.493840 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:22.493676 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2496f073-7bf0-48af-b091-be6b964117ca-kube-api-access-vpbmf podName:2496f073-7bf0-48af-b091-be6b964117ca nodeName:}" failed. No retries permitted until 2026-10-30 21:22:23.493658121 +0000 UTC m=+4.167826987 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-vpbmf" (UniqueName: "kubernetes.io/projected/2496f073-7bf0-48af-b091-be6b964117ca-kube-api-access-vpbmf") pod "network-check-target-wv464" (UID: "2496f073-7bf0-48af-b091-be6b964117ca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:22:22.801636 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:22.801548 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n2sfk" event={"ID":"3cabcccd-6b1e-4089-a3a6-275558de2bfb","Type":"ContainerStarted","Data":"ed963bfa9bbb231b5afbfba50c8b3914a24934b413908bda4c6f037527000a01"} Oct 30 21:22:22.803246 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:22.803208 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8vqf9" event={"ID":"59571898-f15b-4b8f-b043-0e5af369c39e","Type":"ContainerStarted","Data":"b092a4deb6fd47f6577074ba533fcd9831761318517be661b91db0c2191c919f"} Oct 30 21:22:22.804485 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:22.804455 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8rbsm" event={"ID":"d9f7709e-2664-452f-b70b-f4649675781e","Type":"ContainerStarted","Data":"21dac7bbe759d29d25155018511b5532b1b78d6e336d26183d4d7505e28570ab"} Oct 30 21:22:22.805712 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:22.805689 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mxfq6" event={"ID":"5c3735e3-95ac-4293-8555-38ee9ead0353","Type":"ContainerStarted","Data":"e1b78fb648faa00672c441f6ea61fae944f1bcd635af480835decf0a41cc8f60"} Oct 30 21:22:22.806808 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:22.806787 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" event={"ID":"8ea6188f-9e30-4ab0-b5b3-c99647377e00","Type":"ContainerStarted","Data":"53209f7dc29327037916c256cb46bf6239e5ddba43ea43c634c6eb30c579884f"} Oct 30 21:22:22.808045 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:22.808014 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" event={"ID":"3df036ff-1c00-4ce2-9e86-8acbeedf6c82","Type":"ContainerStarted","Data":"9dc54f963eddd9f65ef3f9797b5563e449fed9af9fc472616366c4b6d508a75e"} Oct 30 21:22:22.809174 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:22.809155 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-75hkd" event={"ID":"454a9788-f96e-453d-a83c-63c550c7d96d","Type":"ContainerStarted","Data":"6b2a31073c7737830fb55f6fe12435407df6541f64cf11219ac235d3ab7db600"} Oct 30 21:22:22.811922 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:22.810499 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvfbv" event={"ID":"aed61d9f-eb67-44ae-a094-a12814e42559","Type":"ContainerStarted","Data":"253902cf7fb7a07a5987fe209537eb45f982a10286b6a15f35eacb7027da2cd8"} Oct 30 21:22:22.811922 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:22.811746 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jtrcb" event={"ID":"6a9841f4-72b0-43fd-8aa7-01393d9a01d1","Type":"ContainerStarted","Data":"953276e6631de9919ab79f85e9cad674e90cb10dc00b94a70fea3bf5cef97545"} Oct 30 21:22:22.868984 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:22.868941 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:17:22 +0000 UTC" deadline="2028-05-03 05:16:50.583934089 +0000 UTC" Oct 30 21:22:22.868984 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:22.868974 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13207h54m27.714963404s" Oct 30 21:22:23.402065 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:23.402015 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83a93696-9f3b-472d-9fc8-92350746df75-metrics-certs\") pod \"network-metrics-daemon-lz8cq\" (UID: \"83a93696-9f3b-472d-9fc8-92350746df75\") " pod="openshift-multus/network-metrics-daemon-lz8cq" Oct 30 21:22:23.402288 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:23.402269 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:22:23.402364 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:23.402348 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83a93696-9f3b-472d-9fc8-92350746df75-metrics-certs podName:83a93696-9f3b-472d-9fc8-92350746df75 nodeName:}" failed. No retries permitted until 2026-10-30 21:22:25.40232899 +0000 UTC m=+6.076497861 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/83a93696-9f3b-472d-9fc8-92350746df75-metrics-certs") pod "network-metrics-daemon-lz8cq" (UID: "83a93696-9f3b-472d-9fc8-92350746df75") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:22:23.502891 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:23.502834 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6c9c626a-52a7-466e-b744-f4bd79112fd9-original-pull-secret\") pod \"global-pull-secret-syncer-6nncz\" (UID: \"6c9c626a-52a7-466e-b744-f4bd79112fd9\") " pod="kube-system/global-pull-secret-syncer-6nncz" Oct 30 21:22:23.503112 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:23.502909 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpbmf\" (UniqueName: \"kubernetes.io/projected/2496f073-7bf0-48af-b091-be6b964117ca-kube-api-access-vpbmf\") pod \"network-check-target-wv464\" (UID: \"2496f073-7bf0-48af-b091-be6b964117ca\") " pod="openshift-network-diagnostics/network-check-target-wv464" Oct 30 21:22:23.503112 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:23.503094 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:22:23.503231 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:23.503115 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:22:23.503231 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:23.503129 2582 projected.go:196] Error preparing data for projected volume kube-api-access-vpbmf for pod openshift-network-diagnostics/network-check-target-wv464: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:22:23.503231 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:23.503193 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2496f073-7bf0-48af-b091-be6b964117ca-kube-api-access-vpbmf podName:2496f073-7bf0-48af-b091-be6b964117ca nodeName:}" failed. No retries permitted until 2026-10-30 21:22:25.503173577 +0000 UTC m=+6.177342463 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-vpbmf" (UniqueName: "kubernetes.io/projected/2496f073-7bf0-48af-b091-be6b964117ca-kube-api-access-vpbmf") pod "network-check-target-wv464" (UID: "2496f073-7bf0-48af-b091-be6b964117ca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:22:23.503393 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:23.503270 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:22:23.503393 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:23.503305 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c9c626a-52a7-466e-b744-f4bd79112fd9-original-pull-secret podName:6c9c626a-52a7-466e-b744-f4bd79112fd9 nodeName:}" failed. No retries permitted until 2026-10-30 21:22:25.503293652 +0000 UTC m=+6.177462533 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6c9c626a-52a7-466e-b744-f4bd79112fd9-original-pull-secret") pod "global-pull-secret-syncer-6nncz" (UID: "6c9c626a-52a7-466e-b744-f4bd79112fd9") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:22:23.790853 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:23.790768 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wv464" Oct 30 21:22:23.791042 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:23.790928 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wv464" podUID="2496f073-7bf0-48af-b091-be6b964117ca" Oct 30 21:22:23.791042 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:23.791026 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lz8cq" Oct 30 21:22:23.791195 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:23.791052 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nncz" Oct 30 21:22:23.791195 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:23.791129 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lz8cq" podUID="83a93696-9f3b-472d-9fc8-92350746df75" Oct 30 21:22:23.791304 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:23.791213 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nncz" podUID="6c9c626a-52a7-466e-b744-f4bd79112fd9" Oct 30 21:22:23.869656 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:23.869575 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:17:22 +0000 UTC" deadline="2028-06-01 20:20:19.763403742 +0000 UTC" Oct 30 21:22:23.869656 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:23.869647 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13918h57m55.893762366s" Oct 30 21:22:25.426566 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:25.426298 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83a93696-9f3b-472d-9fc8-92350746df75-metrics-certs\") pod \"network-metrics-daemon-lz8cq\" (UID: \"83a93696-9f3b-472d-9fc8-92350746df75\") " pod="openshift-multus/network-metrics-daemon-lz8cq" Oct 30 21:22:25.427217 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:25.426703 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:22:25.427217 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:25.426778 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83a93696-9f3b-472d-9fc8-92350746df75-metrics-certs podName:83a93696-9f3b-472d-9fc8-92350746df75 nodeName:}" failed. No retries permitted until 2026-10-30 21:22:29.426754813 +0000 UTC m=+10.100923706 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/83a93696-9f3b-472d-9fc8-92350746df75-metrics-certs") pod "network-metrics-daemon-lz8cq" (UID: "83a93696-9f3b-472d-9fc8-92350746df75") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:22:25.527257 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:25.527214 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6c9c626a-52a7-466e-b744-f4bd79112fd9-original-pull-secret\") pod \"global-pull-secret-syncer-6nncz\" (UID: \"6c9c626a-52a7-466e-b744-f4bd79112fd9\") " pod="kube-system/global-pull-secret-syncer-6nncz" Oct 30 21:22:25.527449 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:25.527277 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpbmf\" (UniqueName: \"kubernetes.io/projected/2496f073-7bf0-48af-b091-be6b964117ca-kube-api-access-vpbmf\") pod \"network-check-target-wv464\" (UID: \"2496f073-7bf0-48af-b091-be6b964117ca\") " pod="openshift-network-diagnostics/network-check-target-wv464" Oct 30 21:22:25.527521 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:25.527491 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:22:25.527521 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:25.527511 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:22:25.527651 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:25.527524 2582 projected.go:196] Error preparing data for projected volume kube-api-access-vpbmf for pod openshift-network-diagnostics/network-check-target-wv464: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:22:25.527651 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:25.527585 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2496f073-7bf0-48af-b091-be6b964117ca-kube-api-access-vpbmf podName:2496f073-7bf0-48af-b091-be6b964117ca nodeName:}" failed. No retries permitted until 2026-10-30 21:22:29.527565953 +0000 UTC m=+10.201734824 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-vpbmf" (UniqueName: "kubernetes.io/projected/2496f073-7bf0-48af-b091-be6b964117ca-kube-api-access-vpbmf") pod "network-check-target-wv464" (UID: "2496f073-7bf0-48af-b091-be6b964117ca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:22:25.527771 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:25.527653 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:22:25.527771 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:25.527691 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c9c626a-52a7-466e-b744-f4bd79112fd9-original-pull-secret podName:6c9c626a-52a7-466e-b744-f4bd79112fd9 nodeName:}" failed. No retries permitted until 2026-10-30 21:22:29.52767722 +0000 UTC m=+10.201846101 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6c9c626a-52a7-466e-b744-f4bd79112fd9-original-pull-secret") pod "global-pull-secret-syncer-6nncz" (UID: "6c9c626a-52a7-466e-b744-f4bd79112fd9") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:22:25.793950 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:25.793456 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nncz" Oct 30 21:22:25.793950 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:25.793601 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nncz" podUID="6c9c626a-52a7-466e-b744-f4bd79112fd9" Oct 30 21:22:25.793950 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:25.793665 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wv464" Oct 30 21:22:25.793950 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:25.793741 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wv464" podUID="2496f073-7bf0-48af-b091-be6b964117ca" Oct 30 21:22:25.793950 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:25.793796 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lz8cq" Oct 30 21:22:25.793950 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:25.793881 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lz8cq" podUID="83a93696-9f3b-472d-9fc8-92350746df75" Oct 30 21:22:25.839483 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:25.839441 2582 generic.go:356] "Generic (PLEG): container finished" podID="ddfcdf71a39fda8bf039a6129472bc5e" containerID="b66a7a5f218ecb9b19b19ec59bd98b65015cc0de8d93e9717506f216ddee69b9" exitCode=0 Oct 30 21:22:25.839660 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:25.839508 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-61.us-west-1.compute.internal" event={"ID":"ddfcdf71a39fda8bf039a6129472bc5e","Type":"ContainerDied","Data":"b66a7a5f218ecb9b19b19ec59bd98b65015cc0de8d93e9717506f216ddee69b9"} Oct 30 21:22:26.862830 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:26.862019 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-61.us-west-1.compute.internal" event={"ID":"f05411f241c17e725b228582bf16173f","Type":"ContainerStarted","Data":"2e14ed1439248491146a102d5629ecd25c0f853e53d5c09e5da28c8e46eb2176"} Oct 30 21:22:26.866853 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:26.866293 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-61.us-west-1.compute.internal" event={"ID":"ddfcdf71a39fda8bf039a6129472bc5e","Type":"ContainerStarted","Data":"45745fa7e9c1e170441829c6d7712210c0a40666f8edadf521bffbbb608a90d9"} Oct 30 21:22:26.872150 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:26.872047 2582 generic.go:356] "Generic (PLEG): container finished" podID="6a9841f4-72b0-43fd-8aa7-01393d9a01d1" containerID="77827c8c81a7fd96b753fc502c45fbbf24ad8d7ce6cf8e753275b4f067d42d90" exitCode=0 Oct 30 21:22:26.872150 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:26.872097 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jtrcb" event={"ID":"6a9841f4-72b0-43fd-8aa7-01393d9a01d1","Type":"ContainerDied","Data":"77827c8c81a7fd96b753fc502c45fbbf24ad8d7ce6cf8e753275b4f067d42d90"} Oct 30 21:22:26.986013 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:26.985917 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-61.us-west-1.compute.internal" podStartSLOduration=4.985898823 podStartE2EDuration="4.985898823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:22:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:26.985487396 +0000 UTC m=+7.659656288" watchObservedRunningTime="2026-10-30 21:22:26.985898823 +0000 UTC m=+7.660067713" Oct 30 21:22:27.109839 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:27.109773 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-61.us-west-1.compute.internal" podStartSLOduration=5.10975021 podStartE2EDuration="5.10975021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:22:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:27.109719409 +0000 UTC m=+7.783888295" watchObservedRunningTime="2026-10-30 21:22:27.10975021 +0000 UTC m=+7.783919131" Oct 30 21:22:27.793387 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:27.792829 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wv464" Oct 30 21:22:27.793387 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:27.793002 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wv464" podUID="2496f073-7bf0-48af-b091-be6b964117ca" Oct 30 21:22:27.793387 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:27.793098 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lz8cq" Oct 30 21:22:27.793387 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:27.793192 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lz8cq" podUID="83a93696-9f3b-472d-9fc8-92350746df75" Oct 30 21:22:27.793387 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:27.793252 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nncz" Oct 30 21:22:27.793387 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:27.793327 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nncz" podUID="6c9c626a-52a7-466e-b744-f4bd79112fd9" Oct 30 21:22:27.878621 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:27.878574 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-75hkd" event={"ID":"454a9788-f96e-453d-a83c-63c550c7d96d","Type":"ContainerStarted","Data":"d23d4069d20cedc848776a1eeca2830d5059e6d3dbaecbfb4fca0b363cf08641"} Oct 30 21:22:28.886567 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:28.886498 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8vqf9" event={"ID":"59571898-f15b-4b8f-b043-0e5af369c39e","Type":"ContainerStarted","Data":"19331f47d01f10cb5e9423404ac8cca2852a796710e78a3c99e498d46f57a809"} Oct 30 21:22:29.011022 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:29.010575 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8vqf9" podStartSLOduration=2.997419145 podStartE2EDuration="9.010558809s" totalImagesPullingTime="6.013139664s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:22:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:29.010170223 +0000 UTC m=+9.684339105" watchObservedRunningTime="2026-10-30 21:22:29.010558809 +0000 UTC m=+9.684727716" Oct 30 21:22:29.011022 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:29.010823 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-75hkd" podStartSLOduration=4.136293484 podStartE2EDuration="9.010814004s" totalImagesPullingTime="4.87452052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:28.003775489 +0000 UTC m=+8.677944406" watchObservedRunningTime="2026-10-30 21:22:29.010814004 +0000 UTC m=+9.684982892" Oct 30 21:22:29.461817 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:29.461760 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83a93696-9f3b-472d-9fc8-92350746df75-metrics-certs\") pod \"network-metrics-daemon-lz8cq\" (UID: \"83a93696-9f3b-472d-9fc8-92350746df75\") " pod="openshift-multus/network-metrics-daemon-lz8cq" Oct 30 21:22:29.462050 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:29.461932 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:22:29.462171 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:29.462052 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83a93696-9f3b-472d-9fc8-92350746df75-metrics-certs podName:83a93696-9f3b-472d-9fc8-92350746df75 nodeName:}" failed. No retries permitted until 2026-10-30 21:22:37.462029873 +0000 UTC m=+18.136198751 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/83a93696-9f3b-472d-9fc8-92350746df75-metrics-certs") pod "network-metrics-daemon-lz8cq" (UID: "83a93696-9f3b-472d-9fc8-92350746df75") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:22:29.563256 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:29.563208 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6c9c626a-52a7-466e-b744-f4bd79112fd9-original-pull-secret\") pod \"global-pull-secret-syncer-6nncz\" (UID: \"6c9c626a-52a7-466e-b744-f4bd79112fd9\") " pod="kube-system/global-pull-secret-syncer-6nncz" Oct 30 21:22:29.563464 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:29.563275 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpbmf\" (UniqueName: \"kubernetes.io/projected/2496f073-7bf0-48af-b091-be6b964117ca-kube-api-access-vpbmf\") pod \"network-check-target-wv464\" (UID: \"2496f073-7bf0-48af-b091-be6b964117ca\") " pod="openshift-network-diagnostics/network-check-target-wv464" Oct 30 21:22:29.563528 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:29.563475 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:22:29.563528 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:29.563500 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:22:29.563528 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:29.563513 2582 projected.go:196] Error preparing data for projected volume kube-api-access-vpbmf for pod openshift-network-diagnostics/network-check-target-wv464: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:22:29.563690 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:29.563564 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2496f073-7bf0-48af-b091-be6b964117ca-kube-api-access-vpbmf podName:2496f073-7bf0-48af-b091-be6b964117ca nodeName:}" failed. No retries permitted until 2026-10-30 21:22:37.563549457 +0000 UTC m=+18.237718322 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-vpbmf" (UniqueName: "kubernetes.io/projected/2496f073-7bf0-48af-b091-be6b964117ca-kube-api-access-vpbmf") pod "network-check-target-wv464" (UID: "2496f073-7bf0-48af-b091-be6b964117ca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:22:29.563690 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:29.563628 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:22:29.563690 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:29.563662 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c9c626a-52a7-466e-b744-f4bd79112fd9-original-pull-secret podName:6c9c626a-52a7-466e-b744-f4bd79112fd9 nodeName:}" failed. No retries permitted until 2026-10-30 21:22:37.563651119 +0000 UTC m=+18.237819987 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6c9c626a-52a7-466e-b744-f4bd79112fd9-original-pull-secret") pod "global-pull-secret-syncer-6nncz" (UID: "6c9c626a-52a7-466e-b744-f4bd79112fd9") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:22:29.795117 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:29.795032 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lz8cq" Oct 30 21:22:29.795280 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:29.795153 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nncz" Oct 30 21:22:29.795350 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:29.795298 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lz8cq" podUID="83a93696-9f3b-472d-9fc8-92350746df75" Oct 30 21:22:29.795350 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:29.795317 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wv464" Oct 30 21:22:29.795476 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:29.795434 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nncz" podUID="6c9c626a-52a7-466e-b744-f4bd79112fd9" Oct 30 21:22:29.795650 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:29.795536 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wv464" podUID="2496f073-7bf0-48af-b091-be6b964117ca" Oct 30 21:22:29.890898 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:29.890860 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvfbv" event={"ID":"aed61d9f-eb67-44ae-a094-a12814e42559","Type":"ContainerStarted","Data":"b3fc897263f780db62c8df937d779009ac46954b3267a5185a26bcd1a6c01f09"} Oct 30 21:22:31.272317 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:31.272241 2582 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 21:22:31.347624 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:31.347586 2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-75hkd" Oct 30 21:22:31.348763 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:31.348734 2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-75hkd" Oct 30 21:22:31.739049 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:31.738942 2582 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:22:31.272266702Z","UUID":"ab885e23-1ce8-4505-8973-d380917f43f1","Handler":null,"Name":"","Endpoint":""} Oct 30 21:22:31.740629 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:31.740606 2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 21:22:31.740629 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:31.740636 2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 21:22:31.790661 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:31.790421 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wv464" Oct 30 21:22:31.790831 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:31.790773 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wv464" podUID="2496f073-7bf0-48af-b091-be6b964117ca" Oct 30 21:22:31.790831 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:31.790471 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nncz" Oct 30 21:22:31.790961 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:31.790875 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nncz" podUID="6c9c626a-52a7-466e-b744-f4bd79112fd9" Oct 30 21:22:31.790961 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:31.790449 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lz8cq" Oct 30 21:22:31.791071 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:31.790991 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lz8cq" podUID="83a93696-9f3b-472d-9fc8-92350746df75" Oct 30 21:22:31.898636 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:31.898544 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvfbv" event={"ID":"aed61d9f-eb67-44ae-a094-a12814e42559","Type":"ContainerStarted","Data":"b462d39a352d7296f0725cd521d7fed1f218585aafc1d6b71d13b08b0a2a61a9"} Oct 30 21:22:31.900641 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:31.900614 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jtrcb" event={"ID":"6a9841f4-72b0-43fd-8aa7-01393d9a01d1","Type":"ContainerStarted","Data":"f73778c6a3166636aaac439843cd6d9cfde39e237e9572eb350e6225c8a999f0"} Oct 30 21:22:31.903317 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:31.903293 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n2sfk" event={"ID":"3cabcccd-6b1e-4089-a3a6-275558de2bfb","Type":"ContainerStarted","Data":"619bebbe5d054f6c8f0afbf46b7681523b48b809bbd4360f12b4424803861ad2"} Oct 30 21:22:31.905673 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:31.905648 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8rbsm" event={"ID":"d9f7709e-2664-452f-b70b-f4649675781e","Type":"ContainerStarted","Data":"15849922c3fb8ef0543cbb9fe8dc9636409850df386efa9f0acaf211408449c1"} Oct 30 21:22:32.140779 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:32.140717 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-n2sfk" podStartSLOduration=3.27137811 podStartE2EDuration="12.140700948s" totalImagesPullingTime="8.869322838s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:22:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:32.140144034 +0000 UTC m=+12.814312920" watchObservedRunningTime="2026-10-30 21:22:32.140700948 +0000 UTC m=+12.814869838" Oct 30 21:22:32.910145 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:32.910113 2582 generic.go:356] "Generic (PLEG): container finished" podID="6a9841f4-72b0-43fd-8aa7-01393d9a01d1" containerID="f73778c6a3166636aaac439843cd6d9cfde39e237e9572eb350e6225c8a999f0" exitCode=0 Oct 30 21:22:32.910605 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:32.910200 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jtrcb" event={"ID":"6a9841f4-72b0-43fd-8aa7-01393d9a01d1","Type":"ContainerDied","Data":"f73778c6a3166636aaac439843cd6d9cfde39e237e9572eb350e6225c8a999f0"} Oct 30 21:22:32.912195 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:32.912165 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mxfq6" event={"ID":"5c3735e3-95ac-4293-8555-38ee9ead0353","Type":"ContainerStarted","Data":"a12535e00ae1b7befa0e957f6754d87e9c939b26bc265a2d5363879262aa818d"} Oct 30 21:22:33.035802 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:33.035701 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-8rbsm" podStartSLOduration=3.770629152 podStartE2EDuration="13.035687078s" totalImagesPullingTime="9.265057926s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:22:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:32.260935007 +0000 UTC m=+12.935103889" watchObservedRunningTime="2026-10-30 21:22:33.035687078 +0000 UTC m=+13.709855963" Oct 30 21:22:33.790948 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:33.790914 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nncz" Oct 30 21:22:33.791128 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:33.790928 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lz8cq" Oct 30 21:22:33.791128 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:33.791065 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nncz" podUID="6c9c626a-52a7-466e-b744-f4bd79112fd9" Oct 30 21:22:33.791247 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:33.791174 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lz8cq" podUID="83a93696-9f3b-472d-9fc8-92350746df75" Oct 30 21:22:33.791444 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:33.791402 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wv464" Oct 30 21:22:33.791515 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:33.791499 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wv464" podUID="2496f073-7bf0-48af-b091-be6b964117ca" Oct 30 21:22:33.915741 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:33.915703 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" event={"ID":"8ea6188f-9e30-4ab0-b5b3-c99647377e00","Type":"ContainerStarted","Data":"7d2da8cf05caf4eab6664c943f7b3d7eacb6e6c71217be632da92dc9d1055e98"} Oct 30 21:22:33.918140 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:33.918090 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvfbv" event={"ID":"aed61d9f-eb67-44ae-a094-a12814e42559","Type":"ContainerStarted","Data":"fd1861b576c96a7ba1bbb53c60e3884576aa9424d0402745d7084bb3320aef1d"} Oct 30 21:22:33.995052 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:33.995016 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-75hkd" Oct 30 21:22:33.995227 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:33.995171 2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:22:33.995948 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:33.995862 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-75hkd" Oct 30 21:22:34.033505 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:34.033451 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" podStartSLOduration=2.556331864 podStartE2EDuration="14.033433816s" totalImagesPullingTime="11.477101952s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:22:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:34.033221355 +0000 UTC m=+14.707390239" watchObservedRunningTime="2026-10-30 21:22:34.033433816 +0000 UTC m=+14.707602703" Oct 30 21:22:34.034032 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:34.033990 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-mxfq6" podStartSLOduration=5.003915635 podStartE2EDuration="14.033976563s" totalImagesPullingTime="9.030060928s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:22:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:33.155807251 +0000 UTC m=+13.829976134" watchObservedRunningTime="2026-10-30 21:22:34.033976563 +0000 UTC m=+14.708145449" Oct 30 21:22:34.268013 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:34.267955 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvfbv" podStartSLOduration=3.569792567 podStartE2EDuration="14.267938381s" totalImagesPullingTime="10.698145814s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:22:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:34.267632343 +0000 UTC m=+14.941801229" watchObservedRunningTime="2026-10-30 21:22:34.267938381 +0000 UTC m=+14.942107267" Oct 30 21:22:34.922409 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:34.922379 2582 generic.go:356] "Generic (PLEG): container finished" podID="6a9841f4-72b0-43fd-8aa7-01393d9a01d1" containerID="d16531bebe49eb52d64188fb04118b94477466fab1cc36a225fa9c904dbb5107" exitCode=0 Oct 30 21:22:34.923021 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:34.922677 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jtrcb" event={"ID":"6a9841f4-72b0-43fd-8aa7-01393d9a01d1","Type":"ContainerDied","Data":"d16531bebe49eb52d64188fb04118b94477466fab1cc36a225fa9c904dbb5107"} Oct 30 21:22:35.790449 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:35.790418 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wv464" Oct 30 21:22:35.790449 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:35.790454 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nncz" Oct 30 21:22:35.790704 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:35.790415 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lz8cq" Oct 30 21:22:35.790704 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:35.790558 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wv464" podUID="2496f073-7bf0-48af-b091-be6b964117ca" Oct 30 21:22:35.790704 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:35.790671 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nncz" podUID="6c9c626a-52a7-466e-b744-f4bd79112fd9" Oct 30 21:22:35.790834 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:35.790747 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lz8cq" podUID="83a93696-9f3b-472d-9fc8-92350746df75" Oct 30 21:22:36.929524 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:36.929249 2582 generic.go:356] "Generic (PLEG): container finished" podID="6a9841f4-72b0-43fd-8aa7-01393d9a01d1" containerID="4196b9d7d2ebc1146ac9a6bd61da349478ea97330dd69555a2d48e2f671b3947" exitCode=0 Oct 30 21:22:36.929524 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:36.929336 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jtrcb" event={"ID":"6a9841f4-72b0-43fd-8aa7-01393d9a01d1","Type":"ContainerDied","Data":"4196b9d7d2ebc1146ac9a6bd61da349478ea97330dd69555a2d48e2f671b3947"} Oct 30 21:22:37.528621 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:37.528596 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83a93696-9f3b-472d-9fc8-92350746df75-metrics-certs\") pod \"network-metrics-daemon-lz8cq\" (UID: \"83a93696-9f3b-472d-9fc8-92350746df75\") " pod="openshift-multus/network-metrics-daemon-lz8cq" Oct 30 21:22:37.528730 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:37.528708 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:22:37.528765 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:37.528754 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83a93696-9f3b-472d-9fc8-92350746df75-metrics-certs podName:83a93696-9f3b-472d-9fc8-92350746df75 nodeName:}" failed. No retries permitted until 2026-10-30 21:22:53.528740398 +0000 UTC m=+34.202909260 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/83a93696-9f3b-472d-9fc8-92350746df75-metrics-certs") pod "network-metrics-daemon-lz8cq" (UID: "83a93696-9f3b-472d-9fc8-92350746df75") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:22:37.629113 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:37.629078 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6c9c626a-52a7-466e-b744-f4bd79112fd9-original-pull-secret\") pod \"global-pull-secret-syncer-6nncz\" (UID: \"6c9c626a-52a7-466e-b744-f4bd79112fd9\") " pod="kube-system/global-pull-secret-syncer-6nncz" Oct 30 21:22:37.629113 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:37.629114 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpbmf\" (UniqueName: \"kubernetes.io/projected/2496f073-7bf0-48af-b091-be6b964117ca-kube-api-access-vpbmf\") pod \"network-check-target-wv464\" (UID: \"2496f073-7bf0-48af-b091-be6b964117ca\") " pod="openshift-network-diagnostics/network-check-target-wv464" Oct 30 21:22:37.629376 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:37.629237 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:22:37.629376 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:37.629237 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:22:37.629376 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:37.629249 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:22:37.629376 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:37.629260 2582 projected.go:196] Error preparing data for projected volume kube-api-access-vpbmf for pod openshift-network-diagnostics/network-check-target-wv464: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:22:37.629376 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:37.629308 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c9c626a-52a7-466e-b744-f4bd79112fd9-original-pull-secret podName:6c9c626a-52a7-466e-b744-f4bd79112fd9 nodeName:}" failed. No retries permitted until 2026-10-30 21:22:53.629289732 +0000 UTC m=+34.303458610 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6c9c626a-52a7-466e-b744-f4bd79112fd9-original-pull-secret") pod "global-pull-secret-syncer-6nncz" (UID: "6c9c626a-52a7-466e-b744-f4bd79112fd9") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:22:37.629376 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:37.629330 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2496f073-7bf0-48af-b091-be6b964117ca-kube-api-access-vpbmf podName:2496f073-7bf0-48af-b091-be6b964117ca nodeName:}" failed. No retries permitted until 2026-10-30 21:22:53.629321664 +0000 UTC m=+34.303490542 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-vpbmf" (UniqueName: "kubernetes.io/projected/2496f073-7bf0-48af-b091-be6b964117ca-kube-api-access-vpbmf") pod "network-check-target-wv464" (UID: "2496f073-7bf0-48af-b091-be6b964117ca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:22:37.791096 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:37.791072 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lz8cq" Oct 30 21:22:37.791269 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:37.791123 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nncz" Oct 30 21:22:37.791269 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:37.791139 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wv464" Oct 30 21:22:37.791269 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:37.791235 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nncz" podUID="6c9c626a-52a7-466e-b744-f4bd79112fd9" Oct 30 21:22:37.791415 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:37.791321 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wv464" podUID="2496f073-7bf0-48af-b091-be6b964117ca" Oct 30 21:22:37.791474 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:37.791428 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lz8cq" podUID="83a93696-9f3b-472d-9fc8-92350746df75" Oct 30 21:22:37.935601 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:37.935428 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j4v8r_3df036ff-1c00-4ce2-9e86-8acbeedf6c82/ovn-acl-logging/0.log" Oct 30 21:22:37.936202 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:37.936009 2582 generic.go:356] "Generic (PLEG): container finished" podID="3df036ff-1c00-4ce2-9e86-8acbeedf6c82" containerID="ef98c47b4672ee0779d14d54ce9aed56e9bd752bc87e11fa1af5e8abd943fc62" exitCode=1 Oct 30 21:22:37.936202 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:37.936069 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" event={"ID":"3df036ff-1c00-4ce2-9e86-8acbeedf6c82","Type":"ContainerStarted","Data":"63c9213f4fe2afbc483f0f0dc77e879f89e825701fb03e197d3cbad0f83daf45"} Oct 30 21:22:37.936202 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:37.936149 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" event={"ID":"3df036ff-1c00-4ce2-9e86-8acbeedf6c82","Type":"ContainerStarted","Data":"7541f67a8183d179e60e3ef58e99fbb84a5470f646ee22c607d33138e03bb657"} Oct 30 21:22:37.936202 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:37.936200 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" event={"ID":"3df036ff-1c00-4ce2-9e86-8acbeedf6c82","Type":"ContainerStarted","Data":"212b58a8880d88f5e10dd9d738d5e6cf36e6d36e7127f81f50f425c7db037151"} Oct 30 21:22:37.936352 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:37.936248 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" event={"ID":"3df036ff-1c00-4ce2-9e86-8acbeedf6c82","Type":"ContainerStarted","Data":"121c08f334a67a926da8ceea6624f1b1e74b6795322d02f74f5799a5a8c3edb9"} Oct 30 21:22:37.936352 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:37.936289 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" event={"ID":"3df036ff-1c00-4ce2-9e86-8acbeedf6c82","Type":"ContainerDied","Data":"ef98c47b4672ee0779d14d54ce9aed56e9bd752bc87e11fa1af5e8abd943fc62"} Oct 30 21:22:37.936352 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:37.936336 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" event={"ID":"3df036ff-1c00-4ce2-9e86-8acbeedf6c82","Type":"ContainerStarted","Data":"b85817f567b235116db3c41b4c05ddfa071fd92747f52bc485bc1c4173a28262"} Oct 30 21:22:38.941305 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:38.941269 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jtrcb" event={"ID":"6a9841f4-72b0-43fd-8aa7-01393d9a01d1","Type":"ContainerStarted","Data":"3ed99be6f4175ed8899e22f63d55c111048760666e8a2b642b31d447c0ac133f"} Oct 30 21:22:39.791714 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:39.791678 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wv464" Oct 30 21:22:39.791919 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:39.791819 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nncz" Oct 30 21:22:39.791919 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:39.791845 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wv464" podUID="2496f073-7bf0-48af-b091-be6b964117ca" Oct 30 21:22:39.792009 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:39.791917 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nncz" podUID="6c9c626a-52a7-466e-b744-f4bd79112fd9" Oct 30 21:22:39.792009 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:39.791940 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lz8cq" Oct 30 21:22:39.792080 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:39.792035 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lz8cq" podUID="83a93696-9f3b-472d-9fc8-92350746df75" Oct 30 21:22:39.946167 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:39.946133 2582 generic.go:356] "Generic (PLEG): container finished" podID="6a9841f4-72b0-43fd-8aa7-01393d9a01d1" containerID="3ed99be6f4175ed8899e22f63d55c111048760666e8a2b642b31d447c0ac133f" exitCode=0 Oct 30 21:22:39.946698 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:39.946211 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jtrcb" event={"ID":"6a9841f4-72b0-43fd-8aa7-01393d9a01d1","Type":"ContainerDied","Data":"3ed99be6f4175ed8899e22f63d55c111048760666e8a2b642b31d447c0ac133f"} Oct 30 21:22:40.951799 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:40.951769 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j4v8r_3df036ff-1c00-4ce2-9e86-8acbeedf6c82/ovn-acl-logging/0.log" Oct 30 21:22:40.952205 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:40.952162 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" event={"ID":"3df036ff-1c00-4ce2-9e86-8acbeedf6c82","Type":"ContainerStarted","Data":"f247283657b07176448208a1980b733e20b9942cff9ff3f2bc48460a7260998c"} Oct 30 21:22:40.955033 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:40.955007 2582 generic.go:356] "Generic (PLEG): container finished" podID="6a9841f4-72b0-43fd-8aa7-01393d9a01d1" containerID="52ba5ea380e86ac751943c6a3263da3899051502cbcc1801c3280c089f73417e" exitCode=0 Oct 30 21:22:40.955172 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:40.955052 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jtrcb" event={"ID":"6a9841f4-72b0-43fd-8aa7-01393d9a01d1","Type":"ContainerDied","Data":"52ba5ea380e86ac751943c6a3263da3899051502cbcc1801c3280c089f73417e"} Oct 30 21:22:41.790449 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:41.790414 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nncz" Oct 30 21:22:41.790449 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:41.790445 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wv464" Oct 30 21:22:41.790686 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:41.790526 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lz8cq" Oct 30 21:22:41.790686 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:41.790538 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nncz" podUID="6c9c626a-52a7-466e-b744-f4bd79112fd9" Oct 30 21:22:41.790686 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:41.790597 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wv464" podUID="2496f073-7bf0-48af-b091-be6b964117ca" Oct 30 21:22:41.790835 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:41.790724 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lz8cq" podUID="83a93696-9f3b-472d-9fc8-92350746df75" Oct 30 21:22:41.960370 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:41.960338 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jtrcb" event={"ID":"6a9841f4-72b0-43fd-8aa7-01393d9a01d1","Type":"ContainerStarted","Data":"e1ef7937df5ff0ffd78948c09ca7a9307122c776211c50b91a93633c33fc8fc2"} Oct 30 21:22:42.135043 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:42.134988 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jtrcb" podStartSLOduration=9.326656149 podStartE2EDuration="22.134969678s" totalImagesPullingTime="12.808313529s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:22:20 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:42.132192763 +0000 UTC m=+22.806361649" watchObservedRunningTime="2026-10-30 21:22:42.134969678 +0000 UTC m=+22.809138563" Oct 30 21:22:42.966416 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:42.966230 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j4v8r_3df036ff-1c00-4ce2-9e86-8acbeedf6c82/ovn-acl-logging/0.log" Oct 30 21:22:42.967020 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:42.966799 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" event={"ID":"3df036ff-1c00-4ce2-9e86-8acbeedf6c82","Type":"ContainerStarted","Data":"d13cc4a07995a916d3ebd3d645111caa37b5d1acc51271c9e03068b6ff72072b"} Oct 30 21:22:42.967422 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:42.967402 2582 scope.go:122] "RemoveContainer" containerID="ef98c47b4672ee0779d14d54ce9aed56e9bd752bc87e11fa1af5e8abd943fc62" Oct 30 21:22:43.796272 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:43.796242 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nncz" Oct 30 21:22:43.796507 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:43.796368 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nncz" podUID="6c9c626a-52a7-466e-b744-f4bd79112fd9" Oct 30 21:22:43.796598 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:43.796575 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wv464" Oct 30 21:22:43.796732 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:43.796647 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lz8cq" Oct 30 21:22:43.796770 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:43.796742 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lz8cq" podUID="83a93696-9f3b-472d-9fc8-92350746df75" Oct 30 21:22:43.796862 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:43.796815 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wv464" podUID="2496f073-7bf0-48af-b091-be6b964117ca" Oct 30 21:22:43.973142 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:43.973119 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j4v8r_3df036ff-1c00-4ce2-9e86-8acbeedf6c82/ovn-acl-logging/0.log" Oct 30 21:22:43.973551 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:43.973463 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" event={"ID":"3df036ff-1c00-4ce2-9e86-8acbeedf6c82","Type":"ContainerStarted","Data":"71dd9c0b4734533fc4b37db7f760e9773d611eb5fc75af82f4999d3c6f5769b7"} Oct 30 21:22:43.973551 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:43.973530 2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:22:43.973799 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:43.973784 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:43.973888 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:43.973860 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:43.988800 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:43.988776 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:43.988936 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:43.988878 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:44.179034 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:44.178987 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" podStartSLOduration=8.819933101 podStartE2EDuration="24.178973153s" totalImagesPullingTime="15.359040052s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:22:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:44.153610261 +0000 UTC m=+24.827779146" watchObservedRunningTime="2026-10-30 21:22:44.178973153 +0000 UTC m=+24.853142038" Oct 30 21:22:44.764727 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:44.764641 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6nncz"] Oct 30 21:22:44.764883 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:44.764749 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nncz" Oct 30 21:22:44.764883 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:44.764839 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nncz" podUID="6c9c626a-52a7-466e-b744-f4bd79112fd9" Oct 30 21:22:44.765468 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:44.765440 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wv464"] Oct 30 21:22:44.765583 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:44.765539 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wv464" Oct 30 21:22:44.765647 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:44.765630 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wv464" podUID="2496f073-7bf0-48af-b091-be6b964117ca" Oct 30 21:22:44.769292 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:44.769268 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lz8cq"] Oct 30 21:22:44.769395 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:44.769365 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lz8cq" Oct 30 21:22:44.769486 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:44.769469 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lz8cq" podUID="83a93696-9f3b-472d-9fc8-92350746df75" Oct 30 21:22:44.975362 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:44.975332 2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:22:45.791070 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:45.791033 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nncz" Oct 30 21:22:45.791215 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:45.791042 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wv464" Oct 30 21:22:45.791215 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:45.791158 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nncz" podUID="6c9c626a-52a7-466e-b744-f4bd79112fd9" Oct 30 21:22:45.791308 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:45.791211 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wv464" podUID="2496f073-7bf0-48af-b091-be6b964117ca" Oct 30 21:22:45.977000 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:45.976969 2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:22:46.791308 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:46.791276 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lz8cq" Oct 30 21:22:46.791450 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:46.791416 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lz8cq" podUID="83a93696-9f3b-472d-9fc8-92350746df75" Oct 30 21:22:47.791023 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:47.790988 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nncz" Oct 30 21:22:47.791023 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:47.791020 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wv464" Oct 30 21:22:47.791449 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:47.791095 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nncz" podUID="6c9c626a-52a7-466e-b744-f4bd79112fd9" Oct 30 21:22:47.791449 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:47.791228 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wv464" podUID="2496f073-7bf0-48af-b091-be6b964117ca" Oct 30 21:22:48.791300 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:48.791268 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lz8cq" Oct 30 21:22:48.791681 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:48.791390 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lz8cq" podUID="83a93696-9f3b-472d-9fc8-92350746df75" Oct 30 21:22:49.791857 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:49.791606 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wv464" Oct 30 21:22:49.792302 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:49.791667 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nncz" Oct 30 21:22:49.792302 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:49.791975 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wv464" podUID="2496f073-7bf0-48af-b091-be6b964117ca" Oct 30 21:22:49.792302 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:49.792049 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nncz" podUID="6c9c626a-52a7-466e-b744-f4bd79112fd9" Oct 30 21:22:50.064012 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:50.063911 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:50.064167 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:50.064157 2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:22:50.075629 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:50.075600 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:22:50.791329 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:50.791287 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lz8cq" Oct 30 21:22:50.791540 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:50.791431 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lz8cq" podUID="83a93696-9f3b-472d-9fc8-92350746df75" Oct 30 21:22:51.790744 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:51.790706 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wv464" Oct 30 21:22:51.791375 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:51.790709 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nncz" Oct 30 21:22:51.791375 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:51.790813 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wv464" podUID="2496f073-7bf0-48af-b091-be6b964117ca" Oct 30 21:22:51.791375 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:51.790886 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nncz" podUID="6c9c626a-52a7-466e-b744-f4bd79112fd9" Oct 30 21:22:52.790527 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:52.790483 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lz8cq" Oct 30 21:22:52.790733 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:52.790622 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lz8cq" podUID="83a93696-9f3b-472d-9fc8-92350746df75" Oct 30 21:22:53.539098 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:53.539053 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83a93696-9f3b-472d-9fc8-92350746df75-metrics-certs\") pod \"network-metrics-daemon-lz8cq\" (UID: \"83a93696-9f3b-472d-9fc8-92350746df75\") " pod="openshift-multus/network-metrics-daemon-lz8cq" Oct 30 21:22:53.539567 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:53.539195 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:22:53.539567 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:53.539255 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83a93696-9f3b-472d-9fc8-92350746df75-metrics-certs podName:83a93696-9f3b-472d-9fc8-92350746df75 nodeName:}" failed. No retries permitted until 2026-10-30 21:23:25.539239248 +0000 UTC m=+66.213408124 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/83a93696-9f3b-472d-9fc8-92350746df75-metrics-certs") pod "network-metrics-daemon-lz8cq" (UID: "83a93696-9f3b-472d-9fc8-92350746df75") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:22:53.640028 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:53.639988 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6c9c626a-52a7-466e-b744-f4bd79112fd9-original-pull-secret\") pod \"global-pull-secret-syncer-6nncz\" (UID: \"6c9c626a-52a7-466e-b744-f4bd79112fd9\") " pod="kube-system/global-pull-secret-syncer-6nncz" Oct 30 21:22:53.640028 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:53.640031 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpbmf\" (UniqueName: \"kubernetes.io/projected/2496f073-7bf0-48af-b091-be6b964117ca-kube-api-access-vpbmf\") pod \"network-check-target-wv464\" (UID: \"2496f073-7bf0-48af-b091-be6b964117ca\") " pod="openshift-network-diagnostics/network-check-target-wv464" Oct 30 21:22:53.640276 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:53.640140 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:22:53.640276 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:53.640156 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:22:53.640276 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:53.640173 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:22:53.640276 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:53.640186 2582 projected.go:196] Error preparing data for projected volume kube-api-access-vpbmf for pod openshift-network-diagnostics/network-check-target-wv464: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:22:53.640276 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:53.640205 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c9c626a-52a7-466e-b744-f4bd79112fd9-original-pull-secret podName:6c9c626a-52a7-466e-b744-f4bd79112fd9 nodeName:}" failed. No retries permitted until 2026-10-30 21:23:25.640184772 +0000 UTC m=+66.314353635 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6c9c626a-52a7-466e-b744-f4bd79112fd9-original-pull-secret") pod "global-pull-secret-syncer-6nncz" (UID: "6c9c626a-52a7-466e-b744-f4bd79112fd9") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:22:53.640276 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:53.640230 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2496f073-7bf0-48af-b091-be6b964117ca-kube-api-access-vpbmf podName:2496f073-7bf0-48af-b091-be6b964117ca nodeName:}" failed. No retries permitted until 2026-10-30 21:23:25.640217034 +0000 UTC m=+66.314385911 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-vpbmf" (UniqueName: "kubernetes.io/projected/2496f073-7bf0-48af-b091-be6b964117ca-kube-api-access-vpbmf") pod "network-check-target-wv464" (UID: "2496f073-7bf0-48af-b091-be6b964117ca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:22:53.790541 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:53.790447 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nncz" Oct 30 21:22:53.790541 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:53.790483 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wv464" Oct 30 21:22:53.790760 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:53.790557 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nncz" podUID="6c9c626a-52a7-466e-b744-f4bd79112fd9" Oct 30 21:22:53.790760 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:53.790684 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wv464" podUID="2496f073-7bf0-48af-b091-be6b964117ca" Oct 30 21:22:54.791274 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:54.791188 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lz8cq" Oct 30 21:22:54.791750 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:54.791329 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lz8cq" podUID="83a93696-9f3b-472d-9fc8-92350746df75" Oct 30 21:22:55.295215 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:55.295187 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8rbsm_d9f7709e-2664-452f-b70b-f4649675781e/dns-node-resolver/0.log" Oct 30 21:22:55.791061 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:55.791017 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wv464" Oct 30 21:22:55.791061 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:55.791065 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nncz" Oct 30 21:22:55.791318 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:55.791149 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wv464" podUID="2496f073-7bf0-48af-b091-be6b964117ca" Oct 30 21:22:55.791318 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:55.791261 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nncz" podUID="6c9c626a-52a7-466e-b744-f4bd79112fd9" Oct 30 21:22:56.119453 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:56.119376 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8vqf9_59571898-f15b-4b8f-b043-0e5af369c39e/node-ca/0.log" Oct 30 21:22:56.790475 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:56.790436 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lz8cq" Oct 30 21:22:56.790647 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:56.790560 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lz8cq" podUID="83a93696-9f3b-472d-9fc8-92350746df75" Oct 30 21:22:57.791273 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:57.791234 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wv464" Oct 30 21:22:57.791710 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:57.791348 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wv464" podUID="2496f073-7bf0-48af-b091-be6b964117ca" Oct 30 21:22:57.791710 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:57.791399 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nncz" Oct 30 21:22:57.791710 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:57.791494 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nncz" podUID="6c9c626a-52a7-466e-b744-f4bd79112fd9" Oct 30 21:22:58.790791 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:58.790755 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lz8cq" Oct 30 21:22:58.790990 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:58.790864 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lz8cq" podUID="83a93696-9f3b-472d-9fc8-92350746df75" Oct 30 21:22:59.792623 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:59.792585 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nncz" Oct 30 21:22:59.793039 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:59.792717 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nncz" podUID="6c9c626a-52a7-466e-b744-f4bd79112fd9" Oct 30 21:22:59.793039 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:22:59.792759 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wv464" Oct 30 21:22:59.793039 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:22:59.792858 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wv464" podUID="2496f073-7bf0-48af-b091-be6b964117ca" Oct 30 21:23:00.793050 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:00.793007 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lz8cq" Oct 30 21:23:00.793451 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:23:00.793122 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lz8cq" podUID="83a93696-9f3b-472d-9fc8-92350746df75" Oct 30 21:23:01.790963 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:01.790926 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wv464" Oct 30 21:23:01.790963 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:01.790949 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nncz" Oct 30 21:23:01.791187 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:23:01.791066 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wv464" podUID="2496f073-7bf0-48af-b091-be6b964117ca" Oct 30 21:23:01.791187 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:23:01.791169 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nncz" podUID="6c9c626a-52a7-466e-b744-f4bd79112fd9" Oct 30 21:23:02.790643 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:02.790607 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lz8cq" Oct 30 21:23:02.791072 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:23:02.790730 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lz8cq" podUID="83a93696-9f3b-472d-9fc8-92350746df75" Oct 30 21:23:03.791005 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:03.790971 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nncz" Oct 30 21:23:03.791005 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:03.790973 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wv464" Oct 30 21:23:03.791547 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:23:03.791167 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nncz" podUID="6c9c626a-52a7-466e-b744-f4bd79112fd9" Oct 30 21:23:03.791547 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:23:03.791385 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wv464" podUID="2496f073-7bf0-48af-b091-be6b964117ca" Oct 30 21:23:04.791243 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:04.791207 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lz8cq" Oct 30 21:23:04.791714 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:23:04.791347 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lz8cq" podUID="83a93696-9f3b-472d-9fc8-92350746df75" Oct 30 21:23:05.791052 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:05.791013 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nncz" Oct 30 21:23:05.791052 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:05.791040 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wv464" Oct 30 21:23:05.791266 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:23:05.791145 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nncz" podUID="6c9c626a-52a7-466e-b744-f4bd79112fd9" Oct 30 21:23:05.791266 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:23:05.791225 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wv464" podUID="2496f073-7bf0-48af-b091-be6b964117ca" Oct 30 21:23:06.790333 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:06.790299 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lz8cq" Oct 30 21:23:06.790527 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:23:06.790429 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lz8cq" podUID="83a93696-9f3b-472d-9fc8-92350746df75" Oct 30 21:23:07.790650 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:07.790611 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wv464" Oct 30 21:23:07.791065 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:07.790666 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nncz" Oct 30 21:23:07.791065 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:23:07.790753 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wv464" podUID="2496f073-7bf0-48af-b091-be6b964117ca" Oct 30 21:23:07.791065 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:23:07.790851 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nncz" podUID="6c9c626a-52a7-466e-b744-f4bd79112fd9" Oct 30 21:23:08.791305 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:08.791269 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lz8cq" Oct 30 21:23:08.791690 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:23:08.791395 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lz8cq" podUID="83a93696-9f3b-472d-9fc8-92350746df75" Oct 30 21:23:09.127737 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.127664 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-61.us-west-1.compute.internal" event="NodeReady" Oct 30 21:23:09.127976 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.127784 2582 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 21:23:09.244884 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.244849 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-nrm49"] Oct 30 21:23:09.248921 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.248878 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nrm49" Oct 30 21:23:09.294538 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.294504 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vfrjv"] Oct 30 21:23:09.296506 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.296483 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vfrjv" Oct 30 21:23:09.302917 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.302880 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 21:23:09.303031 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.302928 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 21:23:09.303031 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.302965 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vjpnc\"" Oct 30 21:23:09.308191 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.308170 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nrm49"] Oct 30 21:23:09.333875 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.333848 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-zb9t2"] Oct 30 21:23:09.335535 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.335520 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zb9t2" Oct 30 21:23:09.346182 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.346160 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c2b5bbd1-4e37-4d49-9e28-bdb3c3da309f-config-volume\") pod \"dns-default-nrm49\" (UID: \"c2b5bbd1-4e37-4d49-9e28-bdb3c3da309f\") " pod="openshift-dns/dns-default-nrm49" Oct 30 21:23:09.346285 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.346191 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km8jp\" (UniqueName: \"kubernetes.io/projected/c2b5bbd1-4e37-4d49-9e28-bdb3c3da309f-kube-api-access-km8jp\") pod \"dns-default-nrm49\" (UID: \"c2b5bbd1-4e37-4d49-9e28-bdb3c3da309f\") " pod="openshift-dns/dns-default-nrm49" Oct 30 21:23:09.346322 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.346278 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2b5bbd1-4e37-4d49-9e28-bdb3c3da309f-tmp-dir\") pod \"dns-default-nrm49\" (UID: \"c2b5bbd1-4e37-4d49-9e28-bdb3c3da309f\") " pod="openshift-dns/dns-default-nrm49" Oct 30 21:23:09.346366 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.346320 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c2b5bbd1-4e37-4d49-9e28-bdb3c3da309f-metrics-tls\") pod \"dns-default-nrm49\" (UID: \"c2b5bbd1-4e37-4d49-9e28-bdb3c3da309f\") " pod="openshift-dns/dns-default-nrm49" Oct 30 21:23:09.360099 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.360072 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vfrjv"] Oct 30 21:23:09.403601 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.403568 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 21:23:09.403601 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.403600 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 21:23:09.403864 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.403693 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 21:23:09.403864 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.403735 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 21:23:09.403864 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.403732 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-wjtfq\"" Oct 30 21:23:09.403864 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.403756 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 21:23:09.403864 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.403736 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 21:23:09.403864 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.403778 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 21:23:09.403864 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.403777 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5nrzq\"" Oct 30 21:23:09.422447 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.422416 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zb9t2"] Oct 30 21:23:09.447053 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.447016 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c2b5bbd1-4e37-4d49-9e28-bdb3c3da309f-metrics-tls\") pod \"dns-default-nrm49\" (UID: \"c2b5bbd1-4e37-4d49-9e28-bdb3c3da309f\") " pod="openshift-dns/dns-default-nrm49" Oct 30 21:23:09.447053 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.447053 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/42aedd5a-da26-4be4-8ed8-01fb25f3ee5f-crio-socket\") pod \"insights-runtime-extractor-vfrjv\" (UID: \"42aedd5a-da26-4be4-8ed8-01fb25f3ee5f\") " pod="openshift-insights/insights-runtime-extractor-vfrjv" Oct 30 21:23:09.447258 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.447084 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpccd\" (UniqueName: \"kubernetes.io/projected/74930384-19fd-4c0b-b38f-b2f895f7e6e8-kube-api-access-vpccd\") pod \"ingress-canary-zb9t2\" (UID: \"74930384-19fd-4c0b-b38f-b2f895f7e6e8\") " pod="openshift-ingress-canary/ingress-canary-zb9t2" Oct 30 21:23:09.447258 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.447147 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c2b5bbd1-4e37-4d49-9e28-bdb3c3da309f-config-volume\") pod \"dns-default-nrm49\" (UID: \"c2b5bbd1-4e37-4d49-9e28-bdb3c3da309f\") " pod="openshift-dns/dns-default-nrm49" Oct 30 21:23:09.447258 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.447229 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/74930384-19fd-4c0b-b38f-b2f895f7e6e8-cert\") pod \"ingress-canary-zb9t2\" (UID: \"74930384-19fd-4c0b-b38f-b2f895f7e6e8\") " pod="openshift-ingress-canary/ingress-canary-zb9t2" Oct 30 21:23:09.447387 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.447261 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km8jp\" (UniqueName: \"kubernetes.io/projected/c2b5bbd1-4e37-4d49-9e28-bdb3c3da309f-kube-api-access-km8jp\") pod \"dns-default-nrm49\" (UID: \"c2b5bbd1-4e37-4d49-9e28-bdb3c3da309f\") " pod="openshift-dns/dns-default-nrm49" Oct 30 21:23:09.447387 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.447290 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/42aedd5a-da26-4be4-8ed8-01fb25f3ee5f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vfrjv\" (UID: \"42aedd5a-da26-4be4-8ed8-01fb25f3ee5f\") " pod="openshift-insights/insights-runtime-extractor-vfrjv" Oct 30 21:23:09.447387 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.447360 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/42aedd5a-da26-4be4-8ed8-01fb25f3ee5f-data-volume\") pod \"insights-runtime-extractor-vfrjv\" (UID: \"42aedd5a-da26-4be4-8ed8-01fb25f3ee5f\") " pod="openshift-insights/insights-runtime-extractor-vfrjv" Oct 30 21:23:09.447387 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.447378 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/42aedd5a-da26-4be4-8ed8-01fb25f3ee5f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vfrjv\" (UID: \"42aedd5a-da26-4be4-8ed8-01fb25f3ee5f\") " pod="openshift-insights/insights-runtime-extractor-vfrjv" Oct 30 21:23:09.447560 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.447396 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw8lt\" (UniqueName: \"kubernetes.io/projected/42aedd5a-da26-4be4-8ed8-01fb25f3ee5f-kube-api-access-kw8lt\") pod \"insights-runtime-extractor-vfrjv\" (UID: \"42aedd5a-da26-4be4-8ed8-01fb25f3ee5f\") " pod="openshift-insights/insights-runtime-extractor-vfrjv" Oct 30 21:23:09.447560 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.447418 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2b5bbd1-4e37-4d49-9e28-bdb3c3da309f-tmp-dir\") pod \"dns-default-nrm49\" (UID: \"c2b5bbd1-4e37-4d49-9e28-bdb3c3da309f\") " pod="openshift-dns/dns-default-nrm49" Oct 30 21:23:09.447705 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.447686 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2b5bbd1-4e37-4d49-9e28-bdb3c3da309f-tmp-dir\") pod \"dns-default-nrm49\" (UID: \"c2b5bbd1-4e37-4d49-9e28-bdb3c3da309f\") " pod="openshift-dns/dns-default-nrm49" Oct 30 21:23:09.447842 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.447821 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c2b5bbd1-4e37-4d49-9e28-bdb3c3da309f-config-volume\") pod \"dns-default-nrm49\" (UID: \"c2b5bbd1-4e37-4d49-9e28-bdb3c3da309f\") " pod="openshift-dns/dns-default-nrm49" Oct 30 21:23:09.451141 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.451123 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c2b5bbd1-4e37-4d49-9e28-bdb3c3da309f-metrics-tls\") pod \"dns-default-nrm49\" (UID: \"c2b5bbd1-4e37-4d49-9e28-bdb3c3da309f\") " pod="openshift-dns/dns-default-nrm49" Oct 30 21:23:09.508578 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.508541 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km8jp\" (UniqueName: \"kubernetes.io/projected/c2b5bbd1-4e37-4d49-9e28-bdb3c3da309f-kube-api-access-km8jp\") pod \"dns-default-nrm49\" (UID: \"c2b5bbd1-4e37-4d49-9e28-bdb3c3da309f\") " pod="openshift-dns/dns-default-nrm49" Oct 30 21:23:09.548675 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.548632 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/42aedd5a-da26-4be4-8ed8-01fb25f3ee5f-data-volume\") pod \"insights-runtime-extractor-vfrjv\" (UID: \"42aedd5a-da26-4be4-8ed8-01fb25f3ee5f\") " pod="openshift-insights/insights-runtime-extractor-vfrjv" Oct 30 21:23:09.548675 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.548677 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/42aedd5a-da26-4be4-8ed8-01fb25f3ee5f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vfrjv\" (UID: \"42aedd5a-da26-4be4-8ed8-01fb25f3ee5f\") " pod="openshift-insights/insights-runtime-extractor-vfrjv" Oct 30 21:23:09.548955 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.548702 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw8lt\" (UniqueName: \"kubernetes.io/projected/42aedd5a-da26-4be4-8ed8-01fb25f3ee5f-kube-api-access-kw8lt\") pod \"insights-runtime-extractor-vfrjv\" (UID: \"42aedd5a-da26-4be4-8ed8-01fb25f3ee5f\") " pod="openshift-insights/insights-runtime-extractor-vfrjv" Oct 30 21:23:09.548955 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.548744 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/42aedd5a-da26-4be4-8ed8-01fb25f3ee5f-crio-socket\") pod \"insights-runtime-extractor-vfrjv\" (UID: \"42aedd5a-da26-4be4-8ed8-01fb25f3ee5f\") " pod="openshift-insights/insights-runtime-extractor-vfrjv" Oct 30 21:23:09.548955 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.548779 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpccd\" (UniqueName: \"kubernetes.io/projected/74930384-19fd-4c0b-b38f-b2f895f7e6e8-kube-api-access-vpccd\") pod \"ingress-canary-zb9t2\" (UID: \"74930384-19fd-4c0b-b38f-b2f895f7e6e8\") " pod="openshift-ingress-canary/ingress-canary-zb9t2" Oct 30 21:23:09.548955 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.548808 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/74930384-19fd-4c0b-b38f-b2f895f7e6e8-cert\") pod \"ingress-canary-zb9t2\" (UID: \"74930384-19fd-4c0b-b38f-b2f895f7e6e8\") " pod="openshift-ingress-canary/ingress-canary-zb9t2" Oct 30 21:23:09.548955 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.548835 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/42aedd5a-da26-4be4-8ed8-01fb25f3ee5f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vfrjv\" (UID: \"42aedd5a-da26-4be4-8ed8-01fb25f3ee5f\") " pod="openshift-insights/insights-runtime-extractor-vfrjv" Oct 30 21:23:09.549209 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.549001 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/42aedd5a-da26-4be4-8ed8-01fb25f3ee5f-crio-socket\") pod \"insights-runtime-extractor-vfrjv\" (UID: \"42aedd5a-da26-4be4-8ed8-01fb25f3ee5f\") " pod="openshift-insights/insights-runtime-extractor-vfrjv" Oct 30 21:23:09.549209 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.549057 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/42aedd5a-da26-4be4-8ed8-01fb25f3ee5f-data-volume\") pod \"insights-runtime-extractor-vfrjv\" (UID: \"42aedd5a-da26-4be4-8ed8-01fb25f3ee5f\") " pod="openshift-insights/insights-runtime-extractor-vfrjv" Oct 30 21:23:09.549347 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.549319 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/42aedd5a-da26-4be4-8ed8-01fb25f3ee5f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vfrjv\" (UID: \"42aedd5a-da26-4be4-8ed8-01fb25f3ee5f\") " pod="openshift-insights/insights-runtime-extractor-vfrjv" Oct 30 21:23:09.551154 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.551131 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/42aedd5a-da26-4be4-8ed8-01fb25f3ee5f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vfrjv\" (UID: \"42aedd5a-da26-4be4-8ed8-01fb25f3ee5f\") " pod="openshift-insights/insights-runtime-extractor-vfrjv" Oct 30 21:23:09.551604 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.551583 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/74930384-19fd-4c0b-b38f-b2f895f7e6e8-cert\") pod \"ingress-canary-zb9t2\" (UID: \"74930384-19fd-4c0b-b38f-b2f895f7e6e8\") " pod="openshift-ingress-canary/ingress-canary-zb9t2" Oct 30 21:23:09.559341 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.559317 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nrm49" Oct 30 21:23:09.609958 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.609919 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw8lt\" (UniqueName: \"kubernetes.io/projected/42aedd5a-da26-4be4-8ed8-01fb25f3ee5f-kube-api-access-kw8lt\") pod \"insights-runtime-extractor-vfrjv\" (UID: \"42aedd5a-da26-4be4-8ed8-01fb25f3ee5f\") " pod="openshift-insights/insights-runtime-extractor-vfrjv" Oct 30 21:23:09.610233 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.610208 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpccd\" (UniqueName: \"kubernetes.io/projected/74930384-19fd-4c0b-b38f-b2f895f7e6e8-kube-api-access-vpccd\") pod \"ingress-canary-zb9t2\" (UID: \"74930384-19fd-4c0b-b38f-b2f895f7e6e8\") " pod="openshift-ingress-canary/ingress-canary-zb9t2" Oct 30 21:23:09.644411 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.644381 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zb9t2" Oct 30 21:23:09.794017 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.793982 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nncz" Oct 30 21:23:09.794465 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.794096 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wv464" Oct 30 21:23:09.847107 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.847076 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 21:23:09.847284 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.847145 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 21:23:09.847284 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.847146 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-7rrvk\"" Oct 30 21:23:09.847284 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.847199 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-jj5qw\"" Oct 30 21:23:09.847284 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.847219 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 21:23:09.907348 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.907320 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vfrjv" Oct 30 21:23:09.943325 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:09.943255 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nrm49"] Oct 30 21:23:09.947173 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:23:09.947143 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2b5bbd1_4e37_4d49_9e28_bdb3c3da309f.slice/crio-d5ffffe05af64a1646fb52a59c06c8823e1fb52b56b17d704bf19046a036f3b2 WatchSource:0}: Error finding container d5ffffe05af64a1646fb52a59c06c8823e1fb52b56b17d704bf19046a036f3b2: Status 404 returned error can't find the container with id d5ffffe05af64a1646fb52a59c06c8823e1fb52b56b17d704bf19046a036f3b2 Oct 30 21:23:10.031780 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:10.031748 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nrm49" event={"ID":"c2b5bbd1-4e37-4d49-9e28-bdb3c3da309f","Type":"ContainerStarted","Data":"d5ffffe05af64a1646fb52a59c06c8823e1fb52b56b17d704bf19046a036f3b2"} Oct 30 21:23:10.045740 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:10.045139 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zb9t2"] Oct 30 21:23:10.049336 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:23:10.049305 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74930384_19fd_4c0b_b38f_b2f895f7e6e8.slice/crio-ff15d90480ca95d335939e1f2e01526ffd17ff5dc66f00301b363e9c71972421 WatchSource:0}: Error finding container ff15d90480ca95d335939e1f2e01526ffd17ff5dc66f00301b363e9c71972421: Status 404 returned error can't find the container with id ff15d90480ca95d335939e1f2e01526ffd17ff5dc66f00301b363e9c71972421 Oct 30 21:23:10.286054 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:10.285970 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vfrjv"] Oct 30 21:23:10.289527 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:23:10.289499 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42aedd5a_da26_4be4_8ed8_01fb25f3ee5f.slice/crio-a283ea7030cc46493344d2445e51bba7633bc33dd6cd07a5f47a6bdef80a718b WatchSource:0}: Error finding container a283ea7030cc46493344d2445e51bba7633bc33dd6cd07a5f47a6bdef80a718b: Status 404 returned error can't find the container with id a283ea7030cc46493344d2445e51bba7633bc33dd6cd07a5f47a6bdef80a718b Oct 30 21:23:10.791140 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:10.791100 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lz8cq" Oct 30 21:23:10.844078 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:10.844016 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nr2gm\"" Oct 30 21:23:10.844487 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:10.844191 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 21:23:11.038337 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:11.038297 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vfrjv" event={"ID":"42aedd5a-da26-4be4-8ed8-01fb25f3ee5f","Type":"ContainerStarted","Data":"d636f3cc4e59352658f312b5a99aca0d8d91e7e70465a88f6de5aab8c58690aa"} Oct 30 21:23:11.038528 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:11.038379 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vfrjv" event={"ID":"42aedd5a-da26-4be4-8ed8-01fb25f3ee5f","Type":"ContainerStarted","Data":"a283ea7030cc46493344d2445e51bba7633bc33dd6cd07a5f47a6bdef80a718b"} Oct 30 21:23:11.040163 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:11.040092 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zb9t2" event={"ID":"74930384-19fd-4c0b-b38f-b2f895f7e6e8","Type":"ContainerStarted","Data":"ff15d90480ca95d335939e1f2e01526ffd17ff5dc66f00301b363e9c71972421"} Oct 30 21:23:12.045980 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:12.045933 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vfrjv" event={"ID":"42aedd5a-da26-4be4-8ed8-01fb25f3ee5f","Type":"ContainerStarted","Data":"b4f2ea0ee6ff3ef3cb7b1a605b67dddbf4152a91be6ff8c797e377ec2b6173a4"} Oct 30 21:23:12.054103 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:12.054057 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nrm49" event={"ID":"c2b5bbd1-4e37-4d49-9e28-bdb3c3da309f","Type":"ContainerStarted","Data":"a2f178adeb88bcf3af6ca2c3108383a60148ef43743b0f9e9079ed02f5643825"} Oct 30 21:23:13.060001 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:13.059956 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nrm49" event={"ID":"c2b5bbd1-4e37-4d49-9e28-bdb3c3da309f","Type":"ContainerStarted","Data":"661abd409606c3a38cb5496d24dc16b87d38e79b59d6f8cd0e66ae22d43ba26e"} Oct 30 21:23:13.060463 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:13.060055 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-nrm49" Oct 30 21:23:14.064092 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:14.064057 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vfrjv" event={"ID":"42aedd5a-da26-4be4-8ed8-01fb25f3ee5f","Type":"ContainerStarted","Data":"1b06122ed945f00732e74f30ce79bc5af1c961cf6b1d5184acd9c1080ef30786"} Oct 30 21:23:14.065764 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:14.065735 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zb9t2" event={"ID":"74930384-19fd-4c0b-b38f-b2f895f7e6e8","Type":"ContainerStarted","Data":"43372774a646fa46cd61a368a6a3f9f5792da3430af160b4d5b519bcb0489659"} Oct 30 21:23:14.189084 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:14.189025 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-nrm49" podStartSLOduration=3.208103156 podStartE2EDuration="5.189011011s" totalImagesPullingTime="1.980907855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:13.207225268 +0000 UTC m=+53.881394166" watchObservedRunningTime="2026-10-30 21:23:14.189011011 +0000 UTC m=+54.863179895" Oct 30 21:23:14.189340 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:14.189249 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-vfrjv" podStartSLOduration=2.367801751 podStartE2EDuration="5.189244304s" totalImagesPullingTime="2.821442553s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:23:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:14.188522266 +0000 UTC m=+54.862691151" watchObservedRunningTime="2026-10-30 21:23:14.189244304 +0000 UTC m=+54.863413189" Oct 30 21:23:14.306064 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:14.306010 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-zb9t2" podStartSLOduration=1.5375055629999999 podStartE2EDuration="5.30599462s" totalImagesPullingTime="3.768489057s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:14.305024321 +0000 UTC m=+54.979193219" watchObservedRunningTime="2026-10-30 21:23:14.30599462 +0000 UTC m=+54.980163506" Oct 30 21:23:16.775781 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:16.775744 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7qc5s"] Oct 30 21:23:16.777522 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:16.777507 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7qc5s" Oct 30 21:23:16.879912 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:16.879858 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 30 21:23:16.879912 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:16.879880 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-9qngh\"" Oct 30 21:23:16.880805 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:16.880780 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7qc5s"] Oct 30 21:23:16.909770 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:16.909722 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b282542e-5b91-430d-bd5e-03af2e875952-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-7qc5s\" (UID: \"b282542e-5b91-430d-bd5e-03af2e875952\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7qc5s" Oct 30 21:23:17.010250 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:17.010210 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b282542e-5b91-430d-bd5e-03af2e875952-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-7qc5s\" (UID: \"b282542e-5b91-430d-bd5e-03af2e875952\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7qc5s" Oct 30 21:23:17.012702 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:17.012670 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b282542e-5b91-430d-bd5e-03af2e875952-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-7qc5s\" (UID: \"b282542e-5b91-430d-bd5e-03af2e875952\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7qc5s" Oct 30 21:23:17.087005 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:17.086915 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7qc5s" Oct 30 21:23:17.455781 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:17.455734 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7qc5s"] Oct 30 21:23:17.460048 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:23:17.460018 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb282542e_5b91_430d_bd5e_03af2e875952.slice/crio-16851feff42b6b44a4b4f281e1734b22181590f88d68cd1bb7f73d0622bd9703 WatchSource:0}: Error finding container 16851feff42b6b44a4b4f281e1734b22181590f88d68cd1bb7f73d0622bd9703: Status 404 returned error can't find the container with id 16851feff42b6b44a4b4f281e1734b22181590f88d68cd1bb7f73d0622bd9703 Oct 30 21:23:18.077100 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:18.077061 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7qc5s" event={"ID":"b282542e-5b91-430d-bd5e-03af2e875952","Type":"ContainerStarted","Data":"16851feff42b6b44a4b4f281e1734b22181590f88d68cd1bb7f73d0622bd9703"} Oct 30 21:23:19.082525 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:19.082489 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7qc5s" event={"ID":"b282542e-5b91-430d-bd5e-03af2e875952","Type":"ContainerStarted","Data":"0d1e14531a34eb70a529bfac26f730b3f6b3e53e76ddaa10587d7a66459f805a"} Oct 30 21:23:19.082977 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:19.082704 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7qc5s" Oct 30 21:23:19.087565 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:19.087543 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7qc5s" Oct 30 21:23:19.199613 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:19.199555 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7qc5s" podStartSLOduration=1.8150954110000002 podStartE2EDuration="3.199539587s" totalImagesPullingTime="1.384444176s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:19.199228964 +0000 UTC m=+59.873397850" watchObservedRunningTime="2026-10-30 21:23:19.199539587 +0000 UTC m=+59.873708472" Oct 30 21:23:20.286123 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:20.286082 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75hkd_454a9788-f96e-453d-a83c-63c550c7d96d/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:20.393120 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:20.393084 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-fglqz"] Oct 30 21:23:20.395134 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:20.395115 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-fglqz" Oct 30 21:23:20.436052 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:20.436013 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a9ca84f8-a6d5-45f4-9137-799e05b12c11-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-fglqz\" (UID: \"a9ca84f8-a6d5-45f4-9137-799e05b12c11\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fglqz" Oct 30 21:23:20.436052 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:20.436055 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9ca84f8-a6d5-45f4-9137-799e05b12c11-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-fglqz\" (UID: \"a9ca84f8-a6d5-45f4-9137-799e05b12c11\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fglqz" Oct 30 21:23:20.436282 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:20.436140 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbpln\" (UniqueName: \"kubernetes.io/projected/a9ca84f8-a6d5-45f4-9137-799e05b12c11-kube-api-access-nbpln\") pod \"prometheus-operator-9cd89b74c-fglqz\" (UID: \"a9ca84f8-a6d5-45f4-9137-799e05b12c11\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fglqz" Oct 30 21:23:20.436282 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:20.436199 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a9ca84f8-a6d5-45f4-9137-799e05b12c11-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-fglqz\" (UID: \"a9ca84f8-a6d5-45f4-9137-799e05b12c11\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fglqz" Oct 30 21:23:20.448328 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:20.448293 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 21:23:20.498394 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:20.498357 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 21:23:20.498394 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:20.498408 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-598dq\"" Oct 30 21:23:20.498630 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:20.498461 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 21:23:20.498630 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:20.498484 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 21:23:20.498630 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:20.498554 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 21:23:20.499489 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:20.499473 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-fglqz"] Oct 30 21:23:20.536920 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:20.536827 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a9ca84f8-a6d5-45f4-9137-799e05b12c11-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-fglqz\" (UID: \"a9ca84f8-a6d5-45f4-9137-799e05b12c11\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fglqz" Oct 30 21:23:20.536920 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:20.536874 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9ca84f8-a6d5-45f4-9137-799e05b12c11-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-fglqz\" (UID: \"a9ca84f8-a6d5-45f4-9137-799e05b12c11\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fglqz" Oct 30 21:23:20.537092 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:20.537006 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbpln\" (UniqueName: \"kubernetes.io/projected/a9ca84f8-a6d5-45f4-9137-799e05b12c11-kube-api-access-nbpln\") pod \"prometheus-operator-9cd89b74c-fglqz\" (UID: \"a9ca84f8-a6d5-45f4-9137-799e05b12c11\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fglqz" Oct 30 21:23:20.537092 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:20.537082 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a9ca84f8-a6d5-45f4-9137-799e05b12c11-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-fglqz\" (UID: \"a9ca84f8-a6d5-45f4-9137-799e05b12c11\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fglqz" Oct 30 21:23:20.537231 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:23:20.537211 2582 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Oct 30 21:23:20.537312 ip-10-0-143-61 kubenswrapper[2582]: E1030 21:23:20.537299 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9ca84f8-a6d5-45f4-9137-799e05b12c11-prometheus-operator-tls podName:a9ca84f8-a6d5-45f4-9137-799e05b12c11 nodeName:}" failed. No retries permitted until 2026-10-30 21:23:21.037278898 +0000 UTC m=+61.711447772 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/a9ca84f8-a6d5-45f4-9137-799e05b12c11-prometheus-operator-tls") pod "prometheus-operator-9cd89b74c-fglqz" (UID: "a9ca84f8-a6d5-45f4-9137-799e05b12c11") : secret "prometheus-operator-tls" not found Oct 30 21:23:20.537584 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:20.537563 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9ca84f8-a6d5-45f4-9137-799e05b12c11-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-fglqz\" (UID: \"a9ca84f8-a6d5-45f4-9137-799e05b12c11\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fglqz" Oct 30 21:23:20.539366 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:20.539349 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a9ca84f8-a6d5-45f4-9137-799e05b12c11-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-fglqz\" (UID: \"a9ca84f8-a6d5-45f4-9137-799e05b12c11\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fglqz" Oct 30 21:23:20.603634 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:20.603595 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbpln\" (UniqueName: \"kubernetes.io/projected/a9ca84f8-a6d5-45f4-9137-799e05b12c11-kube-api-access-nbpln\") pod \"prometheus-operator-9cd89b74c-fglqz\" (UID: \"a9ca84f8-a6d5-45f4-9137-799e05b12c11\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fglqz" Oct 30 21:23:21.040733 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:21.040686 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a9ca84f8-a6d5-45f4-9137-799e05b12c11-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-fglqz\" (UID: \"a9ca84f8-a6d5-45f4-9137-799e05b12c11\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fglqz" Oct 30 21:23:21.043069 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:21.043047 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a9ca84f8-a6d5-45f4-9137-799e05b12c11-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-fglqz\" (UID: \"a9ca84f8-a6d5-45f4-9137-799e05b12c11\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fglqz" Oct 30 21:23:21.305616 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:21.305527 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-fglqz" Oct 30 21:23:21.679914 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:21.679863 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-fglqz"] Oct 30 21:23:21.711764 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:23:21.711732 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9ca84f8_a6d5_45f4_9137_799e05b12c11.slice/crio-b4c99a19e2a768b0995b24beda1a0329650dd8b1feed61600397baf6c1a0abe2 WatchSource:0}: Error finding container b4c99a19e2a768b0995b24beda1a0329650dd8b1feed61600397baf6c1a0abe2: Status 404 returned error can't find the container with id b4c99a19e2a768b0995b24beda1a0329650dd8b1feed61600397baf6c1a0abe2 Oct 30 21:23:22.096773 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:22.096683 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-fglqz" event={"ID":"a9ca84f8-a6d5-45f4-9137-799e05b12c11","Type":"ContainerStarted","Data":"b4c99a19e2a768b0995b24beda1a0329650dd8b1feed61600397baf6c1a0abe2"} Oct 30 21:23:23.068517 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:23.068440 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-nrm49" Oct 30 21:23:23.361872 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:23.361787 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-rq2vl"] Oct 30 21:23:23.363803 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:23.363783 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-rq2vl" Oct 30 21:23:23.416455 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:23.416415 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 21:23:23.416455 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:23.416433 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 21:23:23.416647 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:23.416534 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-82qgx\"" Oct 30 21:23:23.424691 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:23.424663 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-rq2vl"] Oct 30 21:23:23.457073 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:23.457028 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2dbd490f-d94c-4f12-9106-a12117deda9d-tmp\") pod \"downloads-6dd789c8b7-rq2vl\" (UID: \"2dbd490f-d94c-4f12-9106-a12117deda9d\") " pod="openshift-console/downloads-6dd789c8b7-rq2vl" Oct 30 21:23:23.457244 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:23.457169 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2xzl\" (UniqueName: \"kubernetes.io/projected/2dbd490f-d94c-4f12-9106-a12117deda9d-kube-api-access-d2xzl\") pod \"downloads-6dd789c8b7-rq2vl\" (UID: \"2dbd490f-d94c-4f12-9106-a12117deda9d\") " pod="openshift-console/downloads-6dd789c8b7-rq2vl" Oct 30 21:23:23.557793 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:23.557769 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2xzl\" (UniqueName: \"kubernetes.io/projected/2dbd490f-d94c-4f12-9106-a12117deda9d-kube-api-access-d2xzl\") pod \"downloads-6dd789c8b7-rq2vl\" (UID: \"2dbd490f-d94c-4f12-9106-a12117deda9d\") " pod="openshift-console/downloads-6dd789c8b7-rq2vl" Oct 30 21:23:23.557906 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:23.557834 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2dbd490f-d94c-4f12-9106-a12117deda9d-tmp\") pod \"downloads-6dd789c8b7-rq2vl\" (UID: \"2dbd490f-d94c-4f12-9106-a12117deda9d\") " pod="openshift-console/downloads-6dd789c8b7-rq2vl" Oct 30 21:23:23.558190 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:23.558171 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2dbd490f-d94c-4f12-9106-a12117deda9d-tmp\") pod \"downloads-6dd789c8b7-rq2vl\" (UID: \"2dbd490f-d94c-4f12-9106-a12117deda9d\") " pod="openshift-console/downloads-6dd789c8b7-rq2vl" Oct 30 21:23:23.619934 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:23.619890 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2xzl\" (UniqueName: \"kubernetes.io/projected/2dbd490f-d94c-4f12-9106-a12117deda9d-kube-api-access-d2xzl\") pod \"downloads-6dd789c8b7-rq2vl\" (UID: \"2dbd490f-d94c-4f12-9106-a12117deda9d\") " pod="openshift-console/downloads-6dd789c8b7-rq2vl" Oct 30 21:23:23.675491 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:23.675456 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-rq2vl" Oct 30 21:23:24.092316 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:24.092282 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-rq2vl"] Oct 30 21:23:24.095436 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:23:24.095405 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dbd490f_d94c_4f12_9106_a12117deda9d.slice/crio-2a6c6abcf5dc8b26d1d6d6f868f70a22fb169f9647327bab20ca6c3646432656 WatchSource:0}: Error finding container 2a6c6abcf5dc8b26d1d6d6f868f70a22fb169f9647327bab20ca6c3646432656: Status 404 returned error can't find the container with id 2a6c6abcf5dc8b26d1d6d6f868f70a22fb169f9647327bab20ca6c3646432656 Oct 30 21:23:24.103278 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:24.103243 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-rq2vl" event={"ID":"2dbd490f-d94c-4f12-9106-a12117deda9d","Type":"ContainerStarted","Data":"2a6c6abcf5dc8b26d1d6d6f868f70a22fb169f9647327bab20ca6c3646432656"} Oct 30 21:23:24.104955 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:24.104934 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-fglqz" event={"ID":"a9ca84f8-a6d5-45f4-9137-799e05b12c11","Type":"ContainerStarted","Data":"63cf11cf9bf9a3b481065cdfcd94d102b35a5aea22cf57d2dca4cad89632b223"} Oct 30 21:23:24.105049 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:24.104975 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-fglqz" event={"ID":"a9ca84f8-a6d5-45f4-9137-799e05b12c11","Type":"ContainerStarted","Data":"2de55bf6649befaa0cd391504bf64884426972da2cca747cbfe3f420713e5c1b"} Oct 30 21:23:25.574181 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:25.574137 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83a93696-9f3b-472d-9fc8-92350746df75-metrics-certs\") pod \"network-metrics-daemon-lz8cq\" (UID: \"83a93696-9f3b-472d-9fc8-92350746df75\") " pod="openshift-multus/network-metrics-daemon-lz8cq" Oct 30 21:23:25.626067 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:25.626032 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 21:23:25.637495 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:25.637462 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83a93696-9f3b-472d-9fc8-92350746df75-metrics-certs\") pod \"network-metrics-daemon-lz8cq\" (UID: \"83a93696-9f3b-472d-9fc8-92350746df75\") " pod="openshift-multus/network-metrics-daemon-lz8cq" Oct 30 21:23:25.674886 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:25.674852 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6c9c626a-52a7-466e-b744-f4bd79112fd9-original-pull-secret\") pod \"global-pull-secret-syncer-6nncz\" (UID: \"6c9c626a-52a7-466e-b744-f4bd79112fd9\") " pod="kube-system/global-pull-secret-syncer-6nncz" Oct 30 21:23:25.675087 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:25.674908 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpbmf\" (UniqueName: \"kubernetes.io/projected/2496f073-7bf0-48af-b091-be6b964117ca-kube-api-access-vpbmf\") pod \"network-check-target-wv464\" (UID: \"2496f073-7bf0-48af-b091-be6b964117ca\") " pod="openshift-network-diagnostics/network-check-target-wv464" Oct 30 21:23:25.727299 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:25.727254 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 21:23:25.774186 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:25.774151 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 21:23:25.777778 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:25.777751 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6c9c626a-52a7-466e-b744-f4bd79112fd9-original-pull-secret\") pod \"global-pull-secret-syncer-6nncz\" (UID: \"6c9c626a-52a7-466e-b744-f4bd79112fd9\") " pod="kube-system/global-pull-secret-syncer-6nncz" Oct 30 21:23:25.787031 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:25.787005 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 21:23:25.800228 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:25.800199 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpbmf\" (UniqueName: \"kubernetes.io/projected/2496f073-7bf0-48af-b091-be6b964117ca-kube-api-access-vpbmf\") pod \"network-check-target-wv464\" (UID: \"2496f073-7bf0-48af-b091-be6b964117ca\") " pod="openshift-network-diagnostics/network-check-target-wv464" Oct 30 21:23:25.855438 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:25.855337 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nr2gm\"" Oct 30 21:23:25.864494 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:25.864449 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lz8cq" Oct 30 21:23:26.057114 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:26.057082 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-jj5qw\"" Oct 30 21:23:26.061484 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:26.061447 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-7rrvk\"" Oct 30 21:23:26.066545 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:26.066515 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nncz" Oct 30 21:23:26.070666 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:26.070642 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wv464" Oct 30 21:23:26.245351 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:26.245287 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-fglqz" podStartSLOduration=4.443611456 podStartE2EDuration="6.245268353s" totalImagesPullingTime="1.801656897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:24.226587351 +0000 UTC m=+64.900756236" watchObservedRunningTime="2026-10-30 21:23:26.245268353 +0000 UTC m=+66.919437239" Oct 30 21:23:26.245650 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:26.245628 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lz8cq"] Oct 30 21:23:26.251045 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:23:26.251013 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83a93696_9f3b_472d_9fc8_92350746df75.slice/crio-313b3e77bdc93435cc744be37f449318264b5b426dbe34048426a475aeadafcb WatchSource:0}: Error finding container 313b3e77bdc93435cc744be37f449318264b5b426dbe34048426a475aeadafcb: Status 404 returned error can't find the container with id 313b3e77bdc93435cc744be37f449318264b5b426dbe34048426a475aeadafcb Oct 30 21:23:26.455647 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:26.455608 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6nncz"] Oct 30 21:23:26.460110 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:26.460006 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wv464"] Oct 30 21:23:26.464122 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:23:26.464095 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2496f073_7bf0_48af_b091_be6b964117ca.slice/crio-7b1cb4f6a870992a27f8a7c72f56ffe89d0e2b5519d6fcad4ff407278af0bc83 WatchSource:0}: Error finding container 7b1cb4f6a870992a27f8a7c72f56ffe89d0e2b5519d6fcad4ff407278af0bc83: Status 404 returned error can't find the container with id 7b1cb4f6a870992a27f8a7c72f56ffe89d0e2b5519d6fcad4ff407278af0bc83 Oct 30 21:23:26.832760 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:26.832677 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-jfpf9"] Oct 30 21:23:26.835768 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:26.835738 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jfpf9" Oct 30 21:23:26.884409 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:26.884374 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ca9f1917-07a8-4ea6-8b59-4f2f99429dfa-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-jfpf9\" (UID: \"ca9f1917-07a8-4ea6-8b59-4f2f99429dfa\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jfpf9" Oct 30 21:23:26.884590 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:26.884415 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ca9f1917-07a8-4ea6-8b59-4f2f99429dfa-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-jfpf9\" (UID: \"ca9f1917-07a8-4ea6-8b59-4f2f99429dfa\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jfpf9" Oct 30 21:23:26.884590 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:26.884462 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca9f1917-07a8-4ea6-8b59-4f2f99429dfa-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-jfpf9\" (UID: \"ca9f1917-07a8-4ea6-8b59-4f2f99429dfa\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jfpf9" Oct 30 21:23:26.884676 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:26.884656 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bghfr\" (UniqueName: \"kubernetes.io/projected/ca9f1917-07a8-4ea6-8b59-4f2f99429dfa-kube-api-access-bghfr\") pod \"openshift-state-metrics-7bd7489957-jfpf9\" (UID: \"ca9f1917-07a8-4ea6-8b59-4f2f99429dfa\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jfpf9" Oct 30 21:23:26.895529 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:26.895490 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-jfpf9"] Oct 30 21:23:26.964148 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:26.964114 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-kqxdq"] Oct 30 21:23:26.966588 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:26.966561 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-kqxdq" Oct 30 21:23:26.973802 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:26.973775 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 21:23:26.973802 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:26.973783 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-4pcbl\"" Oct 30 21:23:26.984993 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:26.984949 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca9f1917-07a8-4ea6-8b59-4f2f99429dfa-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-jfpf9\" (UID: \"ca9f1917-07a8-4ea6-8b59-4f2f99429dfa\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jfpf9" Oct 30 21:23:26.985138 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:26.985012 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/166e5f35-0987-4eef-917a-a4772acba82f-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-kqxdq\" (UID: \"166e5f35-0987-4eef-917a-a4772acba82f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-kqxdq" Oct 30 21:23:26.985138 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:26.985046 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/166e5f35-0987-4eef-917a-a4772acba82f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-kqxdq\" (UID: \"166e5f35-0987-4eef-917a-a4772acba82f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-kqxdq" Oct 30 21:23:26.985138 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:26.985077 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bghfr\" (UniqueName: \"kubernetes.io/projected/ca9f1917-07a8-4ea6-8b59-4f2f99429dfa-kube-api-access-bghfr\") pod \"openshift-state-metrics-7bd7489957-jfpf9\" (UID: \"ca9f1917-07a8-4ea6-8b59-4f2f99429dfa\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jfpf9" Oct 30 21:23:26.985138 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:26.985103 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/166e5f35-0987-4eef-917a-a4772acba82f-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-kqxdq\" (UID: \"166e5f35-0987-4eef-917a-a4772acba82f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-kqxdq" Oct 30 21:23:26.985138 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:26.985128 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/166e5f35-0987-4eef-917a-a4772acba82f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-kqxdq\" (UID: \"166e5f35-0987-4eef-917a-a4772acba82f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-kqxdq" Oct 30 21:23:26.985341 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:26.985151 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-459jm\" (UniqueName: \"kubernetes.io/projected/166e5f35-0987-4eef-917a-a4772acba82f-kube-api-access-459jm\") pod \"kube-state-metrics-5dc8d7d89b-kqxdq\" (UID: \"166e5f35-0987-4eef-917a-a4772acba82f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-kqxdq" Oct 30 21:23:26.985341 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:26.985180 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ca9f1917-07a8-4ea6-8b59-4f2f99429dfa-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-jfpf9\" (UID: \"ca9f1917-07a8-4ea6-8b59-4f2f99429dfa\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jfpf9" Oct 30 21:23:26.985341 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:26.985213 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ca9f1917-07a8-4ea6-8b59-4f2f99429dfa-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-jfpf9\" (UID: \"ca9f1917-07a8-4ea6-8b59-4f2f99429dfa\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jfpf9" Oct 30 21:23:26.985341 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:26.985240 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/166e5f35-0987-4eef-917a-a4772acba82f-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-kqxdq\" (UID: \"166e5f35-0987-4eef-917a-a4772acba82f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-kqxdq" Oct 30 21:23:26.985701 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:26.985679 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca9f1917-07a8-4ea6-8b59-4f2f99429dfa-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-jfpf9\" (UID: \"ca9f1917-07a8-4ea6-8b59-4f2f99429dfa\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jfpf9" Oct 30 21:23:26.988307 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:26.988280 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ca9f1917-07a8-4ea6-8b59-4f2f99429dfa-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-jfpf9\" (UID: \"ca9f1917-07a8-4ea6-8b59-4f2f99429dfa\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jfpf9" Oct 30 21:23:27.012355 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:27.012316 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 21:23:27.013603 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:27.013577 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rlt89"] Oct 30 21:23:27.015697 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:27.015677 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rlt89" Oct 30 21:23:27.018982 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:27.018953 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ca9f1917-07a8-4ea6-8b59-4f2f99429dfa-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-jfpf9\" (UID: \"ca9f1917-07a8-4ea6-8b59-4f2f99429dfa\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jfpf9" Oct 30 21:23:27.050231 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:27.050203 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-kqxdq"] Oct 30 21:23:27.074222 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:27.074185 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 21:23:27.074449 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:27.074339 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-mg5d2\"" Oct 30 21:23:27.074958 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:27.074936 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 21:23:27.075212 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:27.075191 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 21:23:27.075314 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:27.075303 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pvd8m\"" Oct 30 21:23:27.075396 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:27.075382 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 21:23:27.078482 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:27.078459 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bghfr\" (UniqueName: \"kubernetes.io/projected/ca9f1917-07a8-4ea6-8b59-4f2f99429dfa-kube-api-access-bghfr\") pod \"openshift-state-metrics-7bd7489957-jfpf9\" (UID: \"ca9f1917-07a8-4ea6-8b59-4f2f99429dfa\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jfpf9" Oct 30 21:23:27.085694 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:27.085616 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-459jm\" (UniqueName: \"kubernetes.io/projected/166e5f35-0987-4eef-917a-a4772acba82f-kube-api-access-459jm\") pod \"kube-state-metrics-5dc8d7d89b-kqxdq\" (UID: \"166e5f35-0987-4eef-917a-a4772acba82f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-kqxdq" Oct 30 21:23:27.085694 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:27.085673 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979-metrics-client-ca\") pod \"node-exporter-rlt89\" (UID: \"1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979\") " pod="openshift-monitoring/node-exporter-rlt89" Oct 30 21:23:27.085882 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:27.085711 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979-node-exporter-accelerators-collector-config\") pod \"node-exporter-rlt89\" (UID: \"1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979\") " pod="openshift-monitoring/node-exporter-rlt89" Oct 30 21:23:27.085882 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:27.085768 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/166e5f35-0987-4eef-917a-a4772acba82f-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-kqxdq\" (UID: \"166e5f35-0987-4eef-917a-a4772acba82f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-kqxdq" Oct 30 21:23:27.085882 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:27.085794 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979-root\") pod \"node-exporter-rlt89\" (UID: \"1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979\") " pod="openshift-monitoring/node-exporter-rlt89" Oct 30 21:23:27.085882 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:27.085828 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rlt89\" (UID: \"1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979\") " pod="openshift-monitoring/node-exporter-rlt89" Oct 30 21:23:27.086107 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:27.085914 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/166e5f35-0987-4eef-917a-a4772acba82f-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-kqxdq\" (UID: \"166e5f35-0987-4eef-917a-a4772acba82f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-kqxdq" Oct 30 21:23:27.086107 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:27.085969 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs2w6\" (UniqueName: \"kubernetes.io/projected/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979-kube-api-access-hs2w6\") pod \"node-exporter-rlt89\" (UID: \"1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979\") " pod="openshift-monitoring/node-exporter-rlt89" Oct 30 21:23:27.086107 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:27.086013 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/166e5f35-0987-4eef-917a-a4772acba82f-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-kqxdq\" (UID: \"166e5f35-0987-4eef-917a-a4772acba82f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-kqxdq" Oct 30 21:23:27.086107 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:27.086039 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/166e5f35-0987-4eef-917a-a4772acba82f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-kqxdq\" (UID: \"166e5f35-0987-4eef-917a-a4772acba82f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-kqxdq" Oct 30 21:23:27.086107 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:27.086067 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979-sys\") pod \"node-exporter-rlt89\" (UID: \"1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979\") " pod="openshift-monitoring/node-exporter-rlt89" Oct 30 21:23:27.086615 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:27.086426 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979-node-exporter-textfile\") pod \"node-exporter-rlt89\" (UID: \"1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979\") " pod="openshift-monitoring/node-exporter-rlt89" Oct 30 21:23:27.086717 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:27.086683 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/166e5f35-0987-4eef-917a-a4772acba82f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-kqxdq\" (UID: \"166e5f35-0987-4eef-917a-a4772acba82f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-kqxdq" Oct 30 21:23:27.086772 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:27.086730 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979-node-exporter-tls\") pod \"node-exporter-rlt89\" (UID: \"1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979\") " pod="openshift-monitoring/node-exporter-rlt89" Oct 30 21:23:27.086772 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:27.086763 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979-node-exporter-wtmp\") pod \"node-exporter-rlt89\" (UID: \"1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979\") " pod="openshift-monitoring/node-exporter-rlt89" Oct 30 21:23:27.086872 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:27.086774 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/166e5f35-0987-4eef-917a-a4772acba82f-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-kqxdq\" (UID: \"166e5f35-0987-4eef-917a-a4772acba82f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-kqxdq" Oct 30 21:23:27.086872 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:27.086823 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/166e5f35-0987-4eef-917a-a4772acba82f-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-kqxdq\" (UID: \"166e5f35-0987-4eef-917a-a4772acba82f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-kqxdq" Oct 30 21:23:27.089114 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:27.089058 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/166e5f35-0987-4eef-917a-a4772acba82f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-kqxdq\" (UID: \"166e5f35-0987-4eef-917a-a4772acba82f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-kqxdq" Oct 30 21:23:27.089806 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:27.089784 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/166e5f35-0987-4eef-917a-a4772acba82f-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-kqxdq\" (UID: \"166e5f35-0987-4eef-917a-a4772acba82f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-kqxdq" Oct 30 21:23:27.103128 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:27.103095 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 21:23:27.103281 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:27.103231 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 21:23:27.107780 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:27.107753 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/166e5f35-0987-4eef-917a-a4772acba82f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-kqxdq\" (UID: \"166e5f35-0987-4eef-917a-a4772acba82f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-kqxdq" Oct 30 21:23:27.118780 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:27.118740 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6nncz" event={"ID":"6c9c626a-52a7-466e-b744-f4bd79112fd9","Type":"ContainerStarted","Data":"b594b27c3c927b7ab46df23306276cbd54084dabd3bf99c51864a383dc490c7d"} Oct 30 21:23:27.120617 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:27.120594 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wv464" event={"ID":"2496f073-7bf0-48af-b091-be6b964117ca","Type":"ContainerStarted","Data":"7b1cb4f6a870992a27f8a7c72f56ffe89d0e2b5519d6fcad4ff407278af0bc83"} Oct 30 21:23:27.121812 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:27.121789 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lz8cq" event={"ID":"83a93696-9f3b-472d-9fc8-92350746df75","Type":"ContainerStarted","Data":"313b3e77bdc93435cc744be37f449318264b5b426dbe34048426a475aeadafcb"} Oct 30 21:23:27.149769 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:27.149728 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jfpf9" Oct 30 21:23:27.152586 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:27.152469 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-459jm\" (UniqueName: \"kubernetes.io/projected/166e5f35-0987-4eef-917a-a4772acba82f-kube-api-access-459jm\") pod \"kube-state-metrics-5dc8d7d89b-kqxdq\" (UID: \"166e5f35-0987-4eef-917a-a4772acba82f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-kqxdq" Oct 30 21:23:27.187676 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:27.187644 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979-node-exporter-accelerators-collector-config\") pod \"node-exporter-rlt89\" (UID: \"1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979\") " pod="openshift-monitoring/node-exporter-rlt89" Oct 30 21:23:27.187816 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:27.187694 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979-root\") pod \"node-exporter-rlt89\" (UID: \"1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979\") " pod="openshift-monitoring/node-exporter-rlt89" Oct 30 21:23:27.187816 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:27.187715 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rlt89\" (UID: \"1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979\") " pod="openshift-monitoring/node-exporter-rlt89" Oct 30 21:23:27.187816 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:27.187736 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs2w6\" (UniqueName: \"kubernetes.io/projected/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979-kube-api-access-hs2w6\") pod \"node-exporter-rlt89\" (UID: \"1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979\") " pod="openshift-monitoring/node-exporter-rlt89" Oct 30 21:23:27.187816 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:27.187757 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979-sys\") pod \"node-exporter-rlt89\" (UID: \"1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979\") " pod="openshift-monitoring/node-exporter-rlt89" Oct 30 21:23:27.187816 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:27.187771 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979-node-exporter-textfile\") pod \"node-exporter-rlt89\" (UID: \"1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979\") " pod="openshift-monitoring/node-exporter-rlt89" Oct 30 21:23:27.187816 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:27.187791 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979-node-exporter-tls\") pod \"node-exporter-rlt89\" (UID: \"1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979\") " pod="openshift-monitoring/node-exporter-rlt89" Oct 30 21:23:27.187816 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:27.187805 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979-node-exporter-wtmp\") pod \"node-exporter-rlt89\" (UID: \"1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979\") " pod="openshift-monitoring/node-exporter-rlt89" Oct 30 21:23:27.188127 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:27.187825 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979-metrics-client-ca\") pod \"node-exporter-rlt89\" (UID: \"1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979\") " pod="openshift-monitoring/node-exporter-rlt89" Oct 30 21:23:27.188281 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:27.188257 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979-node-exporter-textfile\") pod \"node-exporter-rlt89\" (UID: \"1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979\") " pod="openshift-monitoring/node-exporter-rlt89" Oct 30 21:23:27.188361 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:27.188345 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979-sys\") pod \"node-exporter-rlt89\" (UID: \"1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979\") " pod="openshift-monitoring/node-exporter-rlt89" Oct 30 21:23:27.188499 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:27.188473 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979-metrics-client-ca\") pod \"node-exporter-rlt89\" (UID: \"1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979\") " pod="openshift-monitoring/node-exporter-rlt89" Oct 30 21:23:27.188591 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:27.188558 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979-root\") pod \"node-exporter-rlt89\" (UID: \"1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979\") " pod="openshift-monitoring/node-exporter-rlt89" Oct 30 21:23:27.188687 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:27.188665 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979-node-exporter-accelerators-collector-config\") pod \"node-exporter-rlt89\" (UID: \"1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979\") " pod="openshift-monitoring/node-exporter-rlt89" Oct 30 21:23:27.188830 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:27.188811 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979-node-exporter-wtmp\") pod \"node-exporter-rlt89\" (UID: \"1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979\") " pod="openshift-monitoring/node-exporter-rlt89" Oct 30 21:23:27.191231 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:27.191200 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rlt89\" (UID: \"1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979\") " pod="openshift-monitoring/node-exporter-rlt89" Oct 30 21:23:27.191479 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:27.191335 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979-node-exporter-tls\") pod \"node-exporter-rlt89\" (UID: \"1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979\") " pod="openshift-monitoring/node-exporter-rlt89" Oct 30 21:23:27.253431 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:27.253401 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs2w6\" (UniqueName: \"kubernetes.io/projected/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979-kube-api-access-hs2w6\") pod \"node-exporter-rlt89\" (UID: \"1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979\") " pod="openshift-monitoring/node-exporter-rlt89" Oct 30 21:23:27.281044 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:27.281002 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-kqxdq" Oct 30 21:23:27.336290 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:27.336207 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rlt89" Oct 30 21:23:27.351296 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:23:27.351248 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a1e5944_c376_4bf1_b8ea_ae4d1b6a2979.slice/crio-839bfab96e7f8e0962d2a288948ea4e459936b713d5626d73efb3434d0f2d12a WatchSource:0}: Error finding container 839bfab96e7f8e0962d2a288948ea4e459936b713d5626d73efb3434d0f2d12a: Status 404 returned error can't find the container with id 839bfab96e7f8e0962d2a288948ea4e459936b713d5626d73efb3434d0f2d12a Oct 30 21:23:27.540348 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:27.540026 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-jfpf9"] Oct 30 21:23:27.542589 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:23:27.542543 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca9f1917_07a8_4ea6_8b59_4f2f99429dfa.slice/crio-6d53dc7170d4956d3c83148a21dfb0a03609822ffe6b368653e1331804f3df7d WatchSource:0}: Error finding container 6d53dc7170d4956d3c83148a21dfb0a03609822ffe6b368653e1331804f3df7d: Status 404 returned error can't find the container with id 6d53dc7170d4956d3c83148a21dfb0a03609822ffe6b368653e1331804f3df7d Oct 30 21:23:27.697391 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:27.697303 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-kqxdq"] Oct 30 21:23:27.700365 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:23:27.700337 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod166e5f35_0987_4eef_917a_a4772acba82f.slice/crio-381f7b82d8dc76d1ea1c32732ba49780f515bf1815ff83c89f5870338b25d09c WatchSource:0}: Error finding container 381f7b82d8dc76d1ea1c32732ba49780f515bf1815ff83c89f5870338b25d09c: Status 404 returned error can't find the container with id 381f7b82d8dc76d1ea1c32732ba49780f515bf1815ff83c89f5870338b25d09c Oct 30 21:23:28.129793 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:28.129749 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-kqxdq" event={"ID":"166e5f35-0987-4eef-917a-a4772acba82f","Type":"ContainerStarted","Data":"381f7b82d8dc76d1ea1c32732ba49780f515bf1815ff83c89f5870338b25d09c"} Oct 30 21:23:28.132017 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:28.131927 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rlt89" event={"ID":"1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979","Type":"ContainerStarted","Data":"839bfab96e7f8e0962d2a288948ea4e459936b713d5626d73efb3434d0f2d12a"} Oct 30 21:23:28.135715 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:28.135684 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jfpf9" event={"ID":"ca9f1917-07a8-4ea6-8b59-4f2f99429dfa","Type":"ContainerStarted","Data":"abd7f2dd3b453f6bec43bee50da4d3cbe55fa6c80942d1c7fd5960527de1dd13"} Oct 30 21:23:28.135833 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:28.135748 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jfpf9" event={"ID":"ca9f1917-07a8-4ea6-8b59-4f2f99429dfa","Type":"ContainerStarted","Data":"bcfd9e4adbe8b7c4645d8b4be00c03b2d6bf4ae4ed4ddfbff7fd2ac24972c5b2"} Oct 30 21:23:28.135833 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:28.135779 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jfpf9" event={"ID":"ca9f1917-07a8-4ea6-8b59-4f2f99429dfa","Type":"ContainerStarted","Data":"6d53dc7170d4956d3c83148a21dfb0a03609822ffe6b368653e1331804f3df7d"} Oct 30 21:23:29.154726 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.153754 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lz8cq" event={"ID":"83a93696-9f3b-472d-9fc8-92350746df75","Type":"ContainerStarted","Data":"f87dfa98a27b4c410b3cd3ddebb40e614086053a18276b2a1b3edb30ca2262c4"} Oct 30 21:23:29.154726 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.153854 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lz8cq" event={"ID":"83a93696-9f3b-472d-9fc8-92350746df75","Type":"ContainerStarted","Data":"a004ee4565529e9c3c6e00f3bf32bece91486195fe7882138b915799aa7941bc"} Oct 30 21:23:29.268304 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.266396 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 21:23:29.270886 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.269729 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:29.274688 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.274641 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-lz8cq" podStartSLOduration=67.475825731 podStartE2EDuration="1m9.274627182s" totalImagesPullingTime="1.798801451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:29.274383629 +0000 UTC m=+69.948552515" watchObservedRunningTime="2026-10-30 21:23:29.274627182 +0000 UTC m=+69.948796063" Oct 30 21:23:29.304751 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.304715 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-config-out\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:29.304981 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.304759 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:29.304981 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.304794 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw9tn\" (UniqueName: \"kubernetes.io/projected/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-kube-api-access-sw9tn\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:29.304981 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.304818 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:29.304981 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.304847 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:29.304981 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.304874 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:29.305919 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.305415 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:29.305919 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.305464 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:29.305919 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.305501 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:29.305919 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.305560 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-config-volume\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:29.305919 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.305596 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-web-config\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:29.305919 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.305642 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:29.305919 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.305682 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:29.340435 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.340398 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 21:23:29.340592 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.340444 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 21:23:29.340592 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.340505 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 21:23:29.340592 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.340534 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 21:23:29.340592 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.340590 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 21:23:29.340795 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.340609 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 21:23:29.340795 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.340672 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-bdcrp\"" Oct 30 21:23:29.340882 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.340873 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 21:23:29.407832 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.407004 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-config-volume\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:29.407832 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.407115 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-web-config\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:29.407832 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.407156 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:29.407832 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.407194 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:29.407832 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.407224 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-config-out\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:29.407832 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.407248 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:29.407832 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.407314 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw9tn\" (UniqueName: \"kubernetes.io/projected/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-kube-api-access-sw9tn\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:29.407832 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.407340 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:29.407832 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.407367 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:29.407832 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.407394 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:29.407832 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.407419 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:29.407832 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.407439 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:29.407832 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.407454 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:29.409854 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.409439 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:29.410398 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.410307 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-config-volume\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:29.414372 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.414268 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:29.421513 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.420205 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 21:23:29.422675 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.422581 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 21:23:29.428780 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.428691 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:29.433994 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.433916 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:29.434467 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.434399 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-config-out\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:29.435558 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.435342 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:29.436083 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.436050 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:29.436809 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.436774 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:29.437925 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.437849 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-web-config\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:29.439309 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.439225 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:29.440102 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.440059 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 21:23:29.449622 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.449598 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:29.470858 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.470790 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw9tn\" (UniqueName: \"kubernetes.io/projected/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-kube-api-access-sw9tn\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:29.592806 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.592764 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:29.622174 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.622138 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-76bbc756f6-s5wtw"] Oct 30 21:23:29.626527 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.626499 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-76bbc756f6-s5wtw" Oct 30 21:23:29.690091 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.689974 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-76bbc756f6-s5wtw"] Oct 30 21:23:29.710482 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.709957 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttr7d\" (UniqueName: \"kubernetes.io/projected/a597eced-76f6-49e8-8176-2e9332188be3-kube-api-access-ttr7d\") pod \"thanos-querier-76bbc756f6-s5wtw\" (UID: \"a597eced-76f6-49e8-8176-2e9332188be3\") " pod="openshift-monitoring/thanos-querier-76bbc756f6-s5wtw" Oct 30 21:23:29.710482 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.710010 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a597eced-76f6-49e8-8176-2e9332188be3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-76bbc756f6-s5wtw\" (UID: \"a597eced-76f6-49e8-8176-2e9332188be3\") " pod="openshift-monitoring/thanos-querier-76bbc756f6-s5wtw" Oct 30 21:23:29.710482 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.710048 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a597eced-76f6-49e8-8176-2e9332188be3-secret-grpc-tls\") pod \"thanos-querier-76bbc756f6-s5wtw\" (UID: \"a597eced-76f6-49e8-8176-2e9332188be3\") " pod="openshift-monitoring/thanos-querier-76bbc756f6-s5wtw" Oct 30 21:23:29.710482 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.710080 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a597eced-76f6-49e8-8176-2e9332188be3-metrics-client-ca\") pod \"thanos-querier-76bbc756f6-s5wtw\" (UID: \"a597eced-76f6-49e8-8176-2e9332188be3\") " pod="openshift-monitoring/thanos-querier-76bbc756f6-s5wtw" Oct 30 21:23:29.710482 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.710110 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a597eced-76f6-49e8-8176-2e9332188be3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-76bbc756f6-s5wtw\" (UID: \"a597eced-76f6-49e8-8176-2e9332188be3\") " pod="openshift-monitoring/thanos-querier-76bbc756f6-s5wtw" Oct 30 21:23:29.710482 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.710142 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a597eced-76f6-49e8-8176-2e9332188be3-secret-thanos-querier-tls\") pod \"thanos-querier-76bbc756f6-s5wtw\" (UID: \"a597eced-76f6-49e8-8176-2e9332188be3\") " pod="openshift-monitoring/thanos-querier-76bbc756f6-s5wtw" Oct 30 21:23:29.710482 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.710173 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a597eced-76f6-49e8-8176-2e9332188be3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-76bbc756f6-s5wtw\" (UID: \"a597eced-76f6-49e8-8176-2e9332188be3\") " pod="openshift-monitoring/thanos-querier-76bbc756f6-s5wtw" Oct 30 21:23:29.710482 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.710202 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a597eced-76f6-49e8-8176-2e9332188be3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-76bbc756f6-s5wtw\" (UID: \"a597eced-76f6-49e8-8176-2e9332188be3\") " pod="openshift-monitoring/thanos-querier-76bbc756f6-s5wtw" Oct 30 21:23:29.710482 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.710228 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a597eced-76f6-49e8-8176-2e9332188be3-tmp\") pod \"thanos-querier-76bbc756f6-s5wtw\" (UID: \"a597eced-76f6-49e8-8176-2e9332188be3\") " pod="openshift-monitoring/thanos-querier-76bbc756f6-s5wtw" Oct 30 21:23:29.740649 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.740271 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 21:23:29.740649 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.740377 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-lbktx\"" Oct 30 21:23:29.740649 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.740444 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 21:23:29.740649 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.740479 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 21:23:29.740649 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.740516 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 21:23:29.740649 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.740582 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-b7jogd3ihk5l0\"" Oct 30 21:23:29.740649 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.740617 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 21:23:29.810537 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.810506 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a597eced-76f6-49e8-8176-2e9332188be3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-76bbc756f6-s5wtw\" (UID: \"a597eced-76f6-49e8-8176-2e9332188be3\") " pod="openshift-monitoring/thanos-querier-76bbc756f6-s5wtw" Oct 30 21:23:29.810725 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.810546 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a597eced-76f6-49e8-8176-2e9332188be3-secret-grpc-tls\") pod \"thanos-querier-76bbc756f6-s5wtw\" (UID: \"a597eced-76f6-49e8-8176-2e9332188be3\") " pod="openshift-monitoring/thanos-querier-76bbc756f6-s5wtw" Oct 30 21:23:29.810725 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.810571 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a597eced-76f6-49e8-8176-2e9332188be3-metrics-client-ca\") pod \"thanos-querier-76bbc756f6-s5wtw\" (UID: \"a597eced-76f6-49e8-8176-2e9332188be3\") " pod="openshift-monitoring/thanos-querier-76bbc756f6-s5wtw" Oct 30 21:23:29.810725 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.810588 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a597eced-76f6-49e8-8176-2e9332188be3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-76bbc756f6-s5wtw\" (UID: \"a597eced-76f6-49e8-8176-2e9332188be3\") " pod="openshift-monitoring/thanos-querier-76bbc756f6-s5wtw" Oct 30 21:23:29.810725 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.810612 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a597eced-76f6-49e8-8176-2e9332188be3-secret-thanos-querier-tls\") pod \"thanos-querier-76bbc756f6-s5wtw\" (UID: \"a597eced-76f6-49e8-8176-2e9332188be3\") " pod="openshift-monitoring/thanos-querier-76bbc756f6-s5wtw" Oct 30 21:23:29.810725 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.810633 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a597eced-76f6-49e8-8176-2e9332188be3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-76bbc756f6-s5wtw\" (UID: \"a597eced-76f6-49e8-8176-2e9332188be3\") " pod="openshift-monitoring/thanos-querier-76bbc756f6-s5wtw" Oct 30 21:23:29.810725 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.810650 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a597eced-76f6-49e8-8176-2e9332188be3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-76bbc756f6-s5wtw\" (UID: \"a597eced-76f6-49e8-8176-2e9332188be3\") " pod="openshift-monitoring/thanos-querier-76bbc756f6-s5wtw" Oct 30 21:23:29.810725 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.810666 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a597eced-76f6-49e8-8176-2e9332188be3-tmp\") pod \"thanos-querier-76bbc756f6-s5wtw\" (UID: \"a597eced-76f6-49e8-8176-2e9332188be3\") " pod="openshift-monitoring/thanos-querier-76bbc756f6-s5wtw" Oct 30 21:23:29.810725 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.810716 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttr7d\" (UniqueName: \"kubernetes.io/projected/a597eced-76f6-49e8-8176-2e9332188be3-kube-api-access-ttr7d\") pod \"thanos-querier-76bbc756f6-s5wtw\" (UID: \"a597eced-76f6-49e8-8176-2e9332188be3\") " pod="openshift-monitoring/thanos-querier-76bbc756f6-s5wtw" Oct 30 21:23:29.811814 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.811660 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a597eced-76f6-49e8-8176-2e9332188be3-metrics-client-ca\") pod \"thanos-querier-76bbc756f6-s5wtw\" (UID: \"a597eced-76f6-49e8-8176-2e9332188be3\") " pod="openshift-monitoring/thanos-querier-76bbc756f6-s5wtw" Oct 30 21:23:29.811814 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.811783 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a597eced-76f6-49e8-8176-2e9332188be3-tmp\") pod \"thanos-querier-76bbc756f6-s5wtw\" (UID: \"a597eced-76f6-49e8-8176-2e9332188be3\") " pod="openshift-monitoring/thanos-querier-76bbc756f6-s5wtw" Oct 30 21:23:29.814026 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.813963 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a597eced-76f6-49e8-8176-2e9332188be3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-76bbc756f6-s5wtw\" (UID: \"a597eced-76f6-49e8-8176-2e9332188be3\") " pod="openshift-monitoring/thanos-querier-76bbc756f6-s5wtw" Oct 30 21:23:29.816299 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.815937 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a597eced-76f6-49e8-8176-2e9332188be3-secret-thanos-querier-tls\") pod \"thanos-querier-76bbc756f6-s5wtw\" (UID: \"a597eced-76f6-49e8-8176-2e9332188be3\") " pod="openshift-monitoring/thanos-querier-76bbc756f6-s5wtw" Oct 30 21:23:29.816437 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.816392 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a597eced-76f6-49e8-8176-2e9332188be3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-76bbc756f6-s5wtw\" (UID: \"a597eced-76f6-49e8-8176-2e9332188be3\") " pod="openshift-monitoring/thanos-querier-76bbc756f6-s5wtw" Oct 30 21:23:29.817039 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.816952 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a597eced-76f6-49e8-8176-2e9332188be3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-76bbc756f6-s5wtw\" (UID: \"a597eced-76f6-49e8-8176-2e9332188be3\") " pod="openshift-monitoring/thanos-querier-76bbc756f6-s5wtw" Oct 30 21:23:29.819290 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.819238 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a597eced-76f6-49e8-8176-2e9332188be3-secret-grpc-tls\") pod \"thanos-querier-76bbc756f6-s5wtw\" (UID: \"a597eced-76f6-49e8-8176-2e9332188be3\") " pod="openshift-monitoring/thanos-querier-76bbc756f6-s5wtw" Oct 30 21:23:29.820378 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.820354 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a597eced-76f6-49e8-8176-2e9332188be3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-76bbc756f6-s5wtw\" (UID: \"a597eced-76f6-49e8-8176-2e9332188be3\") " pod="openshift-monitoring/thanos-querier-76bbc756f6-s5wtw" Oct 30 21:23:29.876461 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.876359 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttr7d\" (UniqueName: \"kubernetes.io/projected/a597eced-76f6-49e8-8176-2e9332188be3-kube-api-access-ttr7d\") pod \"thanos-querier-76bbc756f6-s5wtw\" (UID: \"a597eced-76f6-49e8-8176-2e9332188be3\") " pod="openshift-monitoring/thanos-querier-76bbc756f6-s5wtw" Oct 30 21:23:29.948546 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:29.948510 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-76bbc756f6-s5wtw" Oct 30 21:23:30.048821 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:30.048490 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 21:23:30.051552 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:23:30.051370 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd96c9c66_7bd6_4ee6_b6e5_1ca97b1dccc1.slice/crio-ac3b2f70532aadcd84f622494b52343b96ea6fa8e52d777db57d766979f199fb WatchSource:0}: Error finding container ac3b2f70532aadcd84f622494b52343b96ea6fa8e52d777db57d766979f199fb: Status 404 returned error can't find the container with id ac3b2f70532aadcd84f622494b52343b96ea6fa8e52d777db57d766979f199fb Oct 30 21:23:30.163354 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:30.163271 2582 generic.go:356] "Generic (PLEG): container finished" podID="1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979" containerID="c98bd75a6f280f92242c8c79084641d1ae66ee4a29442f1d2a97ff7ab1708a01" exitCode=0 Oct 30 21:23:30.163826 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:30.163383 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rlt89" event={"ID":"1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979","Type":"ContainerDied","Data":"c98bd75a6f280f92242c8c79084641d1ae66ee4a29442f1d2a97ff7ab1708a01"} Oct 30 21:23:30.167743 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:30.167614 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1","Type":"ContainerStarted","Data":"ac3b2f70532aadcd84f622494b52343b96ea6fa8e52d777db57d766979f199fb"} Oct 30 21:23:30.179920 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:30.177121 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jfpf9" event={"ID":"ca9f1917-07a8-4ea6-8b59-4f2f99429dfa","Type":"ContainerStarted","Data":"4b74009f86d9b5f9f529f2a5ac1677b1c824c01ccd6f5eb70a15ea00f9526ece"} Oct 30 21:23:30.354924 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:30.354873 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-76bbc756f6-s5wtw"] Oct 30 21:23:30.408884 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:30.408832 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jfpf9" podStartSLOduration=2.322825495 podStartE2EDuration="4.408818982s" totalImagesPullingTime="2.085993487s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:30.407133767 +0000 UTC m=+71.081302654" watchObservedRunningTime="2026-10-30 21:23:30.408818982 +0000 UTC m=+71.082987867" Oct 30 21:23:31.186592 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:31.186459 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rlt89" event={"ID":"1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979","Type":"ContainerStarted","Data":"0ef5f041953a7084068e0ebf3aed052c904b978ea7809d7f2857c4ae1ef1497f"} Oct 30 21:23:31.187679 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:31.187652 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rlt89" event={"ID":"1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979","Type":"ContainerStarted","Data":"8ba6b68db465dc13d765f99ee158264c7fcb230b2ceacb0ca33d214aab2e1418"} Oct 30 21:23:31.189680 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:31.189653 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76bbc756f6-s5wtw" event={"ID":"a597eced-76f6-49e8-8176-2e9332188be3","Type":"ContainerStarted","Data":"2a4f46994282538de689d63e23d77357bf6940b5effeecfdd9ee73ecf275991b"} Oct 30 21:23:31.193480 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:31.193174 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-kqxdq" event={"ID":"166e5f35-0987-4eef-917a-a4772acba82f","Type":"ContainerStarted","Data":"b037cd4602b6398afbe89feb716a63eb447f2af1cfb827688b7945357b5e300a"} Oct 30 21:23:31.193480 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:31.193222 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-kqxdq" event={"ID":"166e5f35-0987-4eef-917a-a4772acba82f","Type":"ContainerStarted","Data":"efd51d467d0beb954b177d8447af4ecd66f68bed6a731f656d6441a2cb706519"} Oct 30 21:23:31.193480 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:31.193248 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-kqxdq" event={"ID":"166e5f35-0987-4eef-917a-a4772acba82f","Type":"ContainerStarted","Data":"8189977277e43f7b3c967c670856d25f09c21d1850a1967b3c448a2cfb256f1c"} Oct 30 21:23:31.310853 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:31.310241 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rlt89" podStartSLOduration=3.233280837 podStartE2EDuration="5.310222471s" totalImagesPullingTime="1.989034784s" totalInitContainerRuntime="87.90685ms" isStatefulPod=true podCreationTimestamp="2026-10-30 21:23:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:31.308805692 +0000 UTC m=+71.982974577" watchObservedRunningTime="2026-10-30 21:23:31.310222471 +0000 UTC m=+71.984391358" Oct 30 21:23:31.432991 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:31.432928 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-kqxdq" podStartSLOduration=2.895728735 podStartE2EDuration="5.432907028s" totalImagesPullingTime="2.537178293s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:31.432506612 +0000 UTC m=+72.106675497" watchObservedRunningTime="2026-10-30 21:23:31.432907028 +0000 UTC m=+72.107075904" Oct 30 21:23:31.648752 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:31.648715 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-2r74x"] Oct 30 21:23:31.652354 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:31.652331 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2r74x" Oct 30 21:23:31.715524 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:31.715484 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-2r74x"] Oct 30 21:23:31.731871 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:31.731833 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbkld\" (UniqueName: \"kubernetes.io/projected/f0d8d720-b6f1-4600-970f-8191318c3e3e-kube-api-access-lbkld\") pod \"monitoring-plugin-77f4f76594-2r74x\" (UID: \"f0d8d720-b6f1-4600-970f-8191318c3e3e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2r74x" Oct 30 21:23:31.732068 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:31.731931 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f0d8d720-b6f1-4600-970f-8191318c3e3e-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-2r74x\" (UID: \"f0d8d720-b6f1-4600-970f-8191318c3e3e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2r74x" Oct 30 21:23:31.738407 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:31.738376 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-zr9sv\"" Oct 30 21:23:31.738576 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:31.738545 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 21:23:31.833373 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:31.833262 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f0d8d720-b6f1-4600-970f-8191318c3e3e-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-2r74x\" (UID: \"f0d8d720-b6f1-4600-970f-8191318c3e3e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2r74x" Oct 30 21:23:31.833373 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:31.833352 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbkld\" (UniqueName: \"kubernetes.io/projected/f0d8d720-b6f1-4600-970f-8191318c3e3e-kube-api-access-lbkld\") pod \"monitoring-plugin-77f4f76594-2r74x\" (UID: \"f0d8d720-b6f1-4600-970f-8191318c3e3e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2r74x" Oct 30 21:23:31.839193 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:31.839160 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f0d8d720-b6f1-4600-970f-8191318c3e3e-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-2r74x\" (UID: \"f0d8d720-b6f1-4600-970f-8191318c3e3e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2r74x" Oct 30 21:23:31.904642 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:31.904556 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbkld\" (UniqueName: \"kubernetes.io/projected/f0d8d720-b6f1-4600-970f-8191318c3e3e-kube-api-access-lbkld\") pod \"monitoring-plugin-77f4f76594-2r74x\" (UID: \"f0d8d720-b6f1-4600-970f-8191318c3e3e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2r74x" Oct 30 21:23:31.915940 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:31.911449 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5dcb975b97-crpzf"] Oct 30 21:23:31.923584 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:31.922355 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5dcb975b97-crpzf" Oct 30 21:23:31.935133 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:31.935101 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/4372da07-b518-42df-bbd7-f148396bba6b-audit-log\") pod \"metrics-server-5dcb975b97-crpzf\" (UID: \"4372da07-b518-42df-bbd7-f148396bba6b\") " pod="openshift-monitoring/metrics-server-5dcb975b97-crpzf" Oct 30 21:23:31.935254 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:31.935150 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9vqk\" (UniqueName: \"kubernetes.io/projected/4372da07-b518-42df-bbd7-f148396bba6b-kube-api-access-x9vqk\") pod \"metrics-server-5dcb975b97-crpzf\" (UID: \"4372da07-b518-42df-bbd7-f148396bba6b\") " pod="openshift-monitoring/metrics-server-5dcb975b97-crpzf" Oct 30 21:23:31.935254 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:31.935181 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/4372da07-b518-42df-bbd7-f148396bba6b-secret-metrics-server-tls\") pod \"metrics-server-5dcb975b97-crpzf\" (UID: \"4372da07-b518-42df-bbd7-f148396bba6b\") " pod="openshift-monitoring/metrics-server-5dcb975b97-crpzf" Oct 30 21:23:31.935254 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:31.935214 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4372da07-b518-42df-bbd7-f148396bba6b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5dcb975b97-crpzf\" (UID: \"4372da07-b518-42df-bbd7-f148396bba6b\") " pod="openshift-monitoring/metrics-server-5dcb975b97-crpzf" Oct 30 21:23:31.935254 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:31.935244 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4372da07-b518-42df-bbd7-f148396bba6b-client-ca-bundle\") pod \"metrics-server-5dcb975b97-crpzf\" (UID: \"4372da07-b518-42df-bbd7-f148396bba6b\") " pod="openshift-monitoring/metrics-server-5dcb975b97-crpzf" Oct 30 21:23:31.935400 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:31.935312 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/4372da07-b518-42df-bbd7-f148396bba6b-secret-metrics-server-client-certs\") pod \"metrics-server-5dcb975b97-crpzf\" (UID: \"4372da07-b518-42df-bbd7-f148396bba6b\") " pod="openshift-monitoring/metrics-server-5dcb975b97-crpzf" Oct 30 21:23:31.935400 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:31.935358 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/4372da07-b518-42df-bbd7-f148396bba6b-metrics-server-audit-profiles\") pod \"metrics-server-5dcb975b97-crpzf\" (UID: \"4372da07-b518-42df-bbd7-f148396bba6b\") " pod="openshift-monitoring/metrics-server-5dcb975b97-crpzf" Oct 30 21:23:31.967401 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:31.967066 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2r74x" Oct 30 21:23:31.973079 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:31.973006 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5dcb975b97-crpzf"] Oct 30 21:23:31.980208 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:31.978924 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 21:23:31.980208 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:31.979698 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 21:23:31.980208 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:31.979825 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-d6wv5\"" Oct 30 21:23:31.980208 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:31.979969 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-ds517nk6l0lju\"" Oct 30 21:23:31.980208 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:31.980064 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 21:23:31.980208 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:31.980133 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 21:23:32.036249 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:32.036213 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/4372da07-b518-42df-bbd7-f148396bba6b-metrics-server-audit-profiles\") pod \"metrics-server-5dcb975b97-crpzf\" (UID: \"4372da07-b518-42df-bbd7-f148396bba6b\") " pod="openshift-monitoring/metrics-server-5dcb975b97-crpzf" Oct 30 21:23:32.036414 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:32.036264 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/4372da07-b518-42df-bbd7-f148396bba6b-audit-log\") pod \"metrics-server-5dcb975b97-crpzf\" (UID: \"4372da07-b518-42df-bbd7-f148396bba6b\") " pod="openshift-monitoring/metrics-server-5dcb975b97-crpzf" Oct 30 21:23:32.036414 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:32.036298 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9vqk\" (UniqueName: \"kubernetes.io/projected/4372da07-b518-42df-bbd7-f148396bba6b-kube-api-access-x9vqk\") pod \"metrics-server-5dcb975b97-crpzf\" (UID: \"4372da07-b518-42df-bbd7-f148396bba6b\") " pod="openshift-monitoring/metrics-server-5dcb975b97-crpzf" Oct 30 21:23:32.036414 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:32.036331 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/4372da07-b518-42df-bbd7-f148396bba6b-secret-metrics-server-tls\") pod \"metrics-server-5dcb975b97-crpzf\" (UID: \"4372da07-b518-42df-bbd7-f148396bba6b\") " pod="openshift-monitoring/metrics-server-5dcb975b97-crpzf" Oct 30 21:23:32.036414 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:32.036370 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4372da07-b518-42df-bbd7-f148396bba6b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5dcb975b97-crpzf\" (UID: \"4372da07-b518-42df-bbd7-f148396bba6b\") " pod="openshift-monitoring/metrics-server-5dcb975b97-crpzf" Oct 30 21:23:32.036414 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:32.036405 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4372da07-b518-42df-bbd7-f148396bba6b-client-ca-bundle\") pod \"metrics-server-5dcb975b97-crpzf\" (UID: \"4372da07-b518-42df-bbd7-f148396bba6b\") " pod="openshift-monitoring/metrics-server-5dcb975b97-crpzf" Oct 30 21:23:32.036562 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:32.036506 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/4372da07-b518-42df-bbd7-f148396bba6b-secret-metrics-server-client-certs\") pod \"metrics-server-5dcb975b97-crpzf\" (UID: \"4372da07-b518-42df-bbd7-f148396bba6b\") " pod="openshift-monitoring/metrics-server-5dcb975b97-crpzf" Oct 30 21:23:32.037866 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:32.037837 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/4372da07-b518-42df-bbd7-f148396bba6b-audit-log\") pod \"metrics-server-5dcb975b97-crpzf\" (UID: \"4372da07-b518-42df-bbd7-f148396bba6b\") " pod="openshift-monitoring/metrics-server-5dcb975b97-crpzf" Oct 30 21:23:32.038811 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:32.038783 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/4372da07-b518-42df-bbd7-f148396bba6b-metrics-server-audit-profiles\") pod \"metrics-server-5dcb975b97-crpzf\" (UID: \"4372da07-b518-42df-bbd7-f148396bba6b\") " pod="openshift-monitoring/metrics-server-5dcb975b97-crpzf" Oct 30 21:23:32.039349 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:32.039326 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4372da07-b518-42df-bbd7-f148396bba6b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5dcb975b97-crpzf\" (UID: \"4372da07-b518-42df-bbd7-f148396bba6b\") " pod="openshift-monitoring/metrics-server-5dcb975b97-crpzf" Oct 30 21:23:32.045799 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:32.045759 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4372da07-b518-42df-bbd7-f148396bba6b-client-ca-bundle\") pod \"metrics-server-5dcb975b97-crpzf\" (UID: \"4372da07-b518-42df-bbd7-f148396bba6b\") " pod="openshift-monitoring/metrics-server-5dcb975b97-crpzf" Oct 30 21:23:32.046113 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:32.045869 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/4372da07-b518-42df-bbd7-f148396bba6b-secret-metrics-server-client-certs\") pod \"metrics-server-5dcb975b97-crpzf\" (UID: \"4372da07-b518-42df-bbd7-f148396bba6b\") " pod="openshift-monitoring/metrics-server-5dcb975b97-crpzf" Oct 30 21:23:32.048656 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:32.048633 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/4372da07-b518-42df-bbd7-f148396bba6b-secret-metrics-server-tls\") pod \"metrics-server-5dcb975b97-crpzf\" (UID: \"4372da07-b518-42df-bbd7-f148396bba6b\") " pod="openshift-monitoring/metrics-server-5dcb975b97-crpzf" Oct 30 21:23:32.108632 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:32.108592 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9vqk\" (UniqueName: \"kubernetes.io/projected/4372da07-b518-42df-bbd7-f148396bba6b-kube-api-access-x9vqk\") pod \"metrics-server-5dcb975b97-crpzf\" (UID: \"4372da07-b518-42df-bbd7-f148396bba6b\") " pod="openshift-monitoring/metrics-server-5dcb975b97-crpzf" Oct 30 21:23:32.238483 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:32.237749 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5dcb975b97-crpzf" Oct 30 21:23:32.387466 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:32.387299 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-2r74x"] Oct 30 21:23:32.391539 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:23:32.391463 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0d8d720_b6f1_4600_970f_8191318c3e3e.slice/crio-26cd448adaaa27c53605b8c3447aadf82556ccb34651e6e39acc6b4688aaf96a WatchSource:0}: Error finding container 26cd448adaaa27c53605b8c3447aadf82556ccb34651e6e39acc6b4688aaf96a: Status 404 returned error can't find the container with id 26cd448adaaa27c53605b8c3447aadf82556ccb34651e6e39acc6b4688aaf96a Oct 30 21:23:32.647572 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:32.647463 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5dcb975b97-crpzf"] Oct 30 21:23:32.651574 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:23:32.651519 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4372da07_b518_42df_bbd7_f148396bba6b.slice/crio-f6a7fe29d8810324e1e8f160b24c936f3766ae93c0e454b9d78eef9f84b05a86 WatchSource:0}: Error finding container f6a7fe29d8810324e1e8f160b24c936f3766ae93c0e454b9d78eef9f84b05a86: Status 404 returned error can't find the container with id f6a7fe29d8810324e1e8f160b24c936f3766ae93c0e454b9d78eef9f84b05a86 Oct 30 21:23:32.854984 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:32.854879 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7c58f6d48-ls9th"] Oct 30 21:23:32.859752 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:32.859726 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7c58f6d48-ls9th" Oct 30 21:23:32.915306 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:32.915271 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 30 21:23:32.915936 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:32.915881 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 30 21:23:32.944327 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:32.943810 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d8f2a8e7-53db-4867-bd38-a8d45b15e40d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7c58f6d48-ls9th\" (UID: \"d8f2a8e7-53db-4867-bd38-a8d45b15e40d\") " pod="openshift-monitoring/telemeter-client-7c58f6d48-ls9th" Oct 30 21:23:32.944327 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:32.943873 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzjnx\" (UniqueName: \"kubernetes.io/projected/d8f2a8e7-53db-4867-bd38-a8d45b15e40d-kube-api-access-dzjnx\") pod \"telemeter-client-7c58f6d48-ls9th\" (UID: \"d8f2a8e7-53db-4867-bd38-a8d45b15e40d\") " pod="openshift-monitoring/telemeter-client-7c58f6d48-ls9th" Oct 30 21:23:32.944327 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:32.943945 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8f2a8e7-53db-4867-bd38-a8d45b15e40d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7c58f6d48-ls9th\" (UID: \"d8f2a8e7-53db-4867-bd38-a8d45b15e40d\") " pod="openshift-monitoring/telemeter-client-7c58f6d48-ls9th" Oct 30 21:23:32.944327 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:32.943986 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d8f2a8e7-53db-4867-bd38-a8d45b15e40d-secret-telemeter-client\") pod \"telemeter-client-7c58f6d48-ls9th\" (UID: \"d8f2a8e7-53db-4867-bd38-a8d45b15e40d\") " pod="openshift-monitoring/telemeter-client-7c58f6d48-ls9th" Oct 30 21:23:32.944327 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:32.944013 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8f2a8e7-53db-4867-bd38-a8d45b15e40d-metrics-client-ca\") pod \"telemeter-client-7c58f6d48-ls9th\" (UID: \"d8f2a8e7-53db-4867-bd38-a8d45b15e40d\") " pod="openshift-monitoring/telemeter-client-7c58f6d48-ls9th" Oct 30 21:23:32.944327 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:32.944043 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d8f2a8e7-53db-4867-bd38-a8d45b15e40d-federate-client-tls\") pod \"telemeter-client-7c58f6d48-ls9th\" (UID: \"d8f2a8e7-53db-4867-bd38-a8d45b15e40d\") " pod="openshift-monitoring/telemeter-client-7c58f6d48-ls9th" Oct 30 21:23:32.944327 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:32.944071 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d8f2a8e7-53db-4867-bd38-a8d45b15e40d-telemeter-client-tls\") pod \"telemeter-client-7c58f6d48-ls9th\" (UID: \"d8f2a8e7-53db-4867-bd38-a8d45b15e40d\") " pod="openshift-monitoring/telemeter-client-7c58f6d48-ls9th" Oct 30 21:23:32.944327 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:32.944112 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8f2a8e7-53db-4867-bd38-a8d45b15e40d-serving-certs-ca-bundle\") pod \"telemeter-client-7c58f6d48-ls9th\" (UID: \"d8f2a8e7-53db-4867-bd38-a8d45b15e40d\") " pod="openshift-monitoring/telemeter-client-7c58f6d48-ls9th" Oct 30 21:23:32.952721 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:32.952688 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 30 21:23:32.953038 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:32.953011 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 30 21:23:32.953038 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:32.953035 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 30 21:23:32.953210 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:32.953057 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 30 21:23:32.953210 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:32.953145 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-d49np\"" Oct 30 21:23:32.966441 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:32.966402 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7c58f6d48-ls9th"] Oct 30 21:23:33.045070 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:33.045015 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8f2a8e7-53db-4867-bd38-a8d45b15e40d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7c58f6d48-ls9th\" (UID: \"d8f2a8e7-53db-4867-bd38-a8d45b15e40d\") " pod="openshift-monitoring/telemeter-client-7c58f6d48-ls9th" Oct 30 21:23:33.045070 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:33.045071 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d8f2a8e7-53db-4867-bd38-a8d45b15e40d-secret-telemeter-client\") pod \"telemeter-client-7c58f6d48-ls9th\" (UID: \"d8f2a8e7-53db-4867-bd38-a8d45b15e40d\") " pod="openshift-monitoring/telemeter-client-7c58f6d48-ls9th" Oct 30 21:23:33.045274 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:33.045103 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8f2a8e7-53db-4867-bd38-a8d45b15e40d-metrics-client-ca\") pod \"telemeter-client-7c58f6d48-ls9th\" (UID: \"d8f2a8e7-53db-4867-bd38-a8d45b15e40d\") " pod="openshift-monitoring/telemeter-client-7c58f6d48-ls9th" Oct 30 21:23:33.045274 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:33.045132 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d8f2a8e7-53db-4867-bd38-a8d45b15e40d-federate-client-tls\") pod \"telemeter-client-7c58f6d48-ls9th\" (UID: \"d8f2a8e7-53db-4867-bd38-a8d45b15e40d\") " pod="openshift-monitoring/telemeter-client-7c58f6d48-ls9th" Oct 30 21:23:33.045274 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:33.045163 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d8f2a8e7-53db-4867-bd38-a8d45b15e40d-telemeter-client-tls\") pod \"telemeter-client-7c58f6d48-ls9th\" (UID: \"d8f2a8e7-53db-4867-bd38-a8d45b15e40d\") " pod="openshift-monitoring/telemeter-client-7c58f6d48-ls9th" Oct 30 21:23:33.045274 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:33.045204 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8f2a8e7-53db-4867-bd38-a8d45b15e40d-serving-certs-ca-bundle\") pod \"telemeter-client-7c58f6d48-ls9th\" (UID: \"d8f2a8e7-53db-4867-bd38-a8d45b15e40d\") " pod="openshift-monitoring/telemeter-client-7c58f6d48-ls9th" Oct 30 21:23:33.045274 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:33.045263 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d8f2a8e7-53db-4867-bd38-a8d45b15e40d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7c58f6d48-ls9th\" (UID: \"d8f2a8e7-53db-4867-bd38-a8d45b15e40d\") " pod="openshift-monitoring/telemeter-client-7c58f6d48-ls9th" Oct 30 21:23:33.045499 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:33.045313 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzjnx\" (UniqueName: \"kubernetes.io/projected/d8f2a8e7-53db-4867-bd38-a8d45b15e40d-kube-api-access-dzjnx\") pod \"telemeter-client-7c58f6d48-ls9th\" (UID: \"d8f2a8e7-53db-4867-bd38-a8d45b15e40d\") " pod="openshift-monitoring/telemeter-client-7c58f6d48-ls9th" Oct 30 21:23:33.047080 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:33.047037 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8f2a8e7-53db-4867-bd38-a8d45b15e40d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7c58f6d48-ls9th\" (UID: \"d8f2a8e7-53db-4867-bd38-a8d45b15e40d\") " pod="openshift-monitoring/telemeter-client-7c58f6d48-ls9th" Oct 30 21:23:33.047080 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:33.047054 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8f2a8e7-53db-4867-bd38-a8d45b15e40d-metrics-client-ca\") pod \"telemeter-client-7c58f6d48-ls9th\" (UID: \"d8f2a8e7-53db-4867-bd38-a8d45b15e40d\") " pod="openshift-monitoring/telemeter-client-7c58f6d48-ls9th" Oct 30 21:23:33.047233 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:33.047126 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8f2a8e7-53db-4867-bd38-a8d45b15e40d-serving-certs-ca-bundle\") pod \"telemeter-client-7c58f6d48-ls9th\" (UID: \"d8f2a8e7-53db-4867-bd38-a8d45b15e40d\") " pod="openshift-monitoring/telemeter-client-7c58f6d48-ls9th" Oct 30 21:23:33.059927 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:33.053844 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d8f2a8e7-53db-4867-bd38-a8d45b15e40d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7c58f6d48-ls9th\" (UID: \"d8f2a8e7-53db-4867-bd38-a8d45b15e40d\") " pod="openshift-monitoring/telemeter-client-7c58f6d48-ls9th" Oct 30 21:23:33.059927 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:33.053941 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d8f2a8e7-53db-4867-bd38-a8d45b15e40d-telemeter-client-tls\") pod \"telemeter-client-7c58f6d48-ls9th\" (UID: \"d8f2a8e7-53db-4867-bd38-a8d45b15e40d\") " pod="openshift-monitoring/telemeter-client-7c58f6d48-ls9th" Oct 30 21:23:33.059927 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:33.054485 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d8f2a8e7-53db-4867-bd38-a8d45b15e40d-secret-telemeter-client\") pod \"telemeter-client-7c58f6d48-ls9th\" (UID: \"d8f2a8e7-53db-4867-bd38-a8d45b15e40d\") " pod="openshift-monitoring/telemeter-client-7c58f6d48-ls9th" Oct 30 21:23:33.059927 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:33.056672 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d8f2a8e7-53db-4867-bd38-a8d45b15e40d-federate-client-tls\") pod \"telemeter-client-7c58f6d48-ls9th\" (UID: \"d8f2a8e7-53db-4867-bd38-a8d45b15e40d\") " pod="openshift-monitoring/telemeter-client-7c58f6d48-ls9th" Oct 30 21:23:33.110261 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:33.110157 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzjnx\" (UniqueName: \"kubernetes.io/projected/d8f2a8e7-53db-4867-bd38-a8d45b15e40d-kube-api-access-dzjnx\") pod \"telemeter-client-7c58f6d48-ls9th\" (UID: \"d8f2a8e7-53db-4867-bd38-a8d45b15e40d\") " pod="openshift-monitoring/telemeter-client-7c58f6d48-ls9th" Oct 30 21:23:33.180718 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:33.180673 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7c58f6d48-ls9th" Oct 30 21:23:33.207650 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:33.207613 2582 generic.go:356] "Generic (PLEG): container finished" podID="d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1" containerID="ffef4be78f8c37e05b78d506ffd53a0b58cd89c523b3cab54930fca37f7ccc6a" exitCode=0 Oct 30 21:23:33.207817 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:33.207730 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1","Type":"ContainerDied","Data":"ffef4be78f8c37e05b78d506ffd53a0b58cd89c523b3cab54930fca37f7ccc6a"} Oct 30 21:23:33.211568 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:33.211530 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wv464" event={"ID":"2496f073-7bf0-48af-b091-be6b964117ca","Type":"ContainerStarted","Data":"aeffa622e9285ca1198d5b99ad751b6c42ebf066a2bdaa53778fac191ad4627a"} Oct 30 21:23:33.212413 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:33.212388 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wv464" Oct 30 21:23:33.214909 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:33.214869 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2r74x" event={"ID":"f0d8d720-b6f1-4600-970f-8191318c3e3e","Type":"ContainerStarted","Data":"26cd448adaaa27c53605b8c3447aadf82556ccb34651e6e39acc6b4688aaf96a"} Oct 30 21:23:33.216678 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:33.216637 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5dcb975b97-crpzf" event={"ID":"4372da07-b518-42df-bbd7-f148396bba6b","Type":"ContainerStarted","Data":"f6a7fe29d8810324e1e8f160b24c936f3766ae93c0e454b9d78eef9f84b05a86"} Oct 30 21:23:33.480347 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:33.478744 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-wv464" podStartSLOduration=67.422190759 podStartE2EDuration="1m13.47872026s" totalImagesPullingTime="6.056529501s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:33.471454244 +0000 UTC m=+74.145623129" watchObservedRunningTime="2026-10-30 21:23:33.47872026 +0000 UTC m=+74.152889146" Oct 30 21:23:33.617992 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:33.616663 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7fbd54bb44-d7zl5"] Oct 30 21:23:33.620926 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:33.620886 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fbd54bb44-d7zl5" Oct 30 21:23:33.652542 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:33.652303 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-console-config\") pod \"console-7fbd54bb44-d7zl5\" (UID: \"30b586ee-8d8a-4e95-b9a3-64f3b09c77e6\") " pod="openshift-console/console-7fbd54bb44-d7zl5" Oct 30 21:23:33.653007 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:33.652943 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-console-oauth-config\") pod \"console-7fbd54bb44-d7zl5\" (UID: \"30b586ee-8d8a-4e95-b9a3-64f3b09c77e6\") " pod="openshift-console/console-7fbd54bb44-d7zl5" Oct 30 21:23:33.653307 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:33.653162 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nk9v\" (UniqueName: \"kubernetes.io/projected/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-kube-api-access-4nk9v\") pod \"console-7fbd54bb44-d7zl5\" (UID: \"30b586ee-8d8a-4e95-b9a3-64f3b09c77e6\") " pod="openshift-console/console-7fbd54bb44-d7zl5" Oct 30 21:23:33.653514 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:33.653446 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-service-ca\") pod \"console-7fbd54bb44-d7zl5\" (UID: \"30b586ee-8d8a-4e95-b9a3-64f3b09c77e6\") " pod="openshift-console/console-7fbd54bb44-d7zl5" Oct 30 21:23:33.653921 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:33.653855 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-tmp\") pod \"console-7fbd54bb44-d7zl5\" (UID: \"30b586ee-8d8a-4e95-b9a3-64f3b09c77e6\") " pod="openshift-console/console-7fbd54bb44-d7zl5" Oct 30 21:23:33.654179 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:33.654110 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-console-serving-cert\") pod \"console-7fbd54bb44-d7zl5\" (UID: \"30b586ee-8d8a-4e95-b9a3-64f3b09c77e6\") " pod="openshift-console/console-7fbd54bb44-d7zl5" Oct 30 21:23:33.654611 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:33.654519 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-oauth-serving-cert\") pod \"console-7fbd54bb44-d7zl5\" (UID: \"30b586ee-8d8a-4e95-b9a3-64f3b09c77e6\") " pod="openshift-console/console-7fbd54bb44-d7zl5" Oct 30 21:23:33.660574 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:33.660451 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7c58f6d48-ls9th"] Oct 30 21:23:33.674605 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:33.674564 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 21:23:33.675165 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:33.675093 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-k65vr\"" Oct 30 21:23:33.675283 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:33.675093 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 21:23:33.675608 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:33.675472 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 21:23:33.675608 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:33.675551 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 21:23:33.691107 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:33.691058 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 21:23:33.700984 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:33.700885 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7fbd54bb44-d7zl5"] Oct 30 21:23:33.757969 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:33.755670 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-console-oauth-config\") pod \"console-7fbd54bb44-d7zl5\" (UID: \"30b586ee-8d8a-4e95-b9a3-64f3b09c77e6\") " pod="openshift-console/console-7fbd54bb44-d7zl5" Oct 30 21:23:33.757969 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:33.755724 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nk9v\" (UniqueName: \"kubernetes.io/projected/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-kube-api-access-4nk9v\") pod \"console-7fbd54bb44-d7zl5\" (UID: \"30b586ee-8d8a-4e95-b9a3-64f3b09c77e6\") " pod="openshift-console/console-7fbd54bb44-d7zl5" Oct 30 21:23:33.757969 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:33.755759 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-service-ca\") pod \"console-7fbd54bb44-d7zl5\" (UID: \"30b586ee-8d8a-4e95-b9a3-64f3b09c77e6\") " pod="openshift-console/console-7fbd54bb44-d7zl5" Oct 30 21:23:33.757969 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:33.755809 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-tmp\") pod \"console-7fbd54bb44-d7zl5\" (UID: \"30b586ee-8d8a-4e95-b9a3-64f3b09c77e6\") " pod="openshift-console/console-7fbd54bb44-d7zl5" Oct 30 21:23:33.757969 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:33.755838 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-console-serving-cert\") pod \"console-7fbd54bb44-d7zl5\" (UID: \"30b586ee-8d8a-4e95-b9a3-64f3b09c77e6\") " pod="openshift-console/console-7fbd54bb44-d7zl5" Oct 30 21:23:33.757969 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:33.755930 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-oauth-serving-cert\") pod \"console-7fbd54bb44-d7zl5\" (UID: \"30b586ee-8d8a-4e95-b9a3-64f3b09c77e6\") " pod="openshift-console/console-7fbd54bb44-d7zl5" Oct 30 21:23:33.757969 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:33.755979 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-console-config\") pod \"console-7fbd54bb44-d7zl5\" (UID: \"30b586ee-8d8a-4e95-b9a3-64f3b09c77e6\") " pod="openshift-console/console-7fbd54bb44-d7zl5" Oct 30 21:23:33.757969 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:33.756763 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-console-config\") pod \"console-7fbd54bb44-d7zl5\" (UID: \"30b586ee-8d8a-4e95-b9a3-64f3b09c77e6\") " pod="openshift-console/console-7fbd54bb44-d7zl5" Oct 30 21:23:33.757969 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:33.757448 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-tmp\") pod \"console-7fbd54bb44-d7zl5\" (UID: \"30b586ee-8d8a-4e95-b9a3-64f3b09c77e6\") " pod="openshift-console/console-7fbd54bb44-d7zl5" Oct 30 21:23:33.758556 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:33.758516 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-service-ca\") pod \"console-7fbd54bb44-d7zl5\" (UID: \"30b586ee-8d8a-4e95-b9a3-64f3b09c77e6\") " pod="openshift-console/console-7fbd54bb44-d7zl5" Oct 30 21:23:33.758626 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:33.758564 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-oauth-serving-cert\") pod \"console-7fbd54bb44-d7zl5\" (UID: \"30b586ee-8d8a-4e95-b9a3-64f3b09c77e6\") " pod="openshift-console/console-7fbd54bb44-d7zl5" Oct 30 21:23:33.760341 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:33.760313 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-console-oauth-config\") pod \"console-7fbd54bb44-d7zl5\" (UID: \"30b586ee-8d8a-4e95-b9a3-64f3b09c77e6\") " pod="openshift-console/console-7fbd54bb44-d7zl5" Oct 30 21:23:33.760848 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:33.760822 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-console-serving-cert\") pod \"console-7fbd54bb44-d7zl5\" (UID: \"30b586ee-8d8a-4e95-b9a3-64f3b09c77e6\") " pod="openshift-console/console-7fbd54bb44-d7zl5" Oct 30 21:23:33.823108 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:33.823064 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nk9v\" (UniqueName: \"kubernetes.io/projected/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-kube-api-access-4nk9v\") pod \"console-7fbd54bb44-d7zl5\" (UID: \"30b586ee-8d8a-4e95-b9a3-64f3b09c77e6\") " pod="openshift-console/console-7fbd54bb44-d7zl5" Oct 30 21:23:33.939525 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:33.939410 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fbd54bb44-d7zl5" Oct 30 21:23:34.224655 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:34.224614 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7c58f6d48-ls9th" event={"ID":"d8f2a8e7-53db-4867-bd38-a8d45b15e40d","Type":"ContainerStarted","Data":"ee9f13f99dd386c9fdec54d4ce3a925258a068ea279cbfbc01ee5a7d8534a773"} Oct 30 21:23:34.373970 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:34.373877 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7fbd54bb44-d7zl5"] Oct 30 21:23:34.391916 ip-10-0-143-61 kubenswrapper[2582]: W1030 21:23:34.391866 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30b586ee_8d8a_4e95_b9a3_64f3b09c77e6.slice/crio-90b5c7445469629806ce6590748554079f89b212ed99dfc9f33c2f39aea5951c WatchSource:0}: Error finding container 90b5c7445469629806ce6590748554079f89b212ed99dfc9f33c2f39aea5951c: Status 404 returned error can't find the container with id 90b5c7445469629806ce6590748554079f89b212ed99dfc9f33c2f39aea5951c Oct 30 21:23:35.236459 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:35.236191 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76bbc756f6-s5wtw" event={"ID":"a597eced-76f6-49e8-8176-2e9332188be3","Type":"ContainerStarted","Data":"f6c58f86a71be7f64e114be60642deae36ec78d8c41cb75e3456786235e4b69c"} Oct 30 21:23:35.236459 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:35.236283 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76bbc756f6-s5wtw" event={"ID":"a597eced-76f6-49e8-8176-2e9332188be3","Type":"ContainerStarted","Data":"357fd04343136a76b7f7618e031716554fe7c3fa5383681f4b5b7e778371b5bd"} Oct 30 21:23:35.236459 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:35.236337 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76bbc756f6-s5wtw" event={"ID":"a597eced-76f6-49e8-8176-2e9332188be3","Type":"ContainerStarted","Data":"f006662f1fd6387131161819dd27e0929e728aabee9874cdb976a40a266f8e6d"} Oct 30 21:23:35.241011 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:35.240874 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fbd54bb44-d7zl5" event={"ID":"30b586ee-8d8a-4e95-b9a3-64f3b09c77e6","Type":"ContainerStarted","Data":"90b5c7445469629806ce6590748554079f89b212ed99dfc9f33c2f39aea5951c"} Oct 30 21:23:36.162544 ip-10-0-143-61 kubenswrapper[2582]: I1030 21:23:36.162407 2582 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:23:36.162527 ip-10-0-143-61 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 21:23:36.182518 ip-10-0-143-61 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 21:23:36.182743 ip-10-0-143-61 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 21:23:36.182808 ip-10-0-143-61 systemd[1]: kubelet.service: Consumed 4.502s CPU time. Oct 30 21:23:36.195643 ip-10-0-143-61 systemd[1]: Starting Kubernetes Kubelet... Oct 30 21:23:36.299231 ip-10-0-143-61 kubenswrapper[5748]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:23:36.299231 ip-10-0-143-61 kubenswrapper[5748]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 21:23:36.299231 ip-10-0-143-61 kubenswrapper[5748]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:23:36.300036 ip-10-0-143-61 kubenswrapper[5748]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:23:36.303543 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303444 5748 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:23:36.303543 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303540 5748 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:23:36.303543 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303545 5748 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:23:36.303543 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303548 5748 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:23:36.303543 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303552 5748 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:23:36.303755 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303555 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:23:36.303755 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303559 5748 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:23:36.303755 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303562 5748 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:23:36.303755 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303565 5748 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:23:36.303755 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303568 5748 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:23:36.303755 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303570 5748 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:23:36.303755 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303573 5748 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:23:36.303755 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303576 5748 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:23:36.303755 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303578 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:23:36.303755 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303581 5748 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:23:36.303755 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303584 5748 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:23:36.303755 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303587 5748 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:23:36.303755 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303589 5748 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:23:36.303755 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303592 5748 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:23:36.303755 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303594 5748 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:23:36.303755 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303597 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:23:36.303755 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303600 5748 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:23:36.303755 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303603 5748 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:23:36.303755 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303605 5748 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:23:36.304224 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303609 5748 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:23:36.304224 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303611 5748 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:23:36.304224 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303613 5748 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:23:36.304224 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303616 5748 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:23:36.304224 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303618 5748 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:23:36.304224 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303621 5748 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:23:36.304224 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303632 5748 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:23:36.304224 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303635 5748 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:23:36.304224 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303638 5748 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:23:36.304224 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303640 5748 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:23:36.304224 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303643 5748 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:23:36.304224 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303646 5748 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:23:36.304224 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303650 5748 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:23:36.304224 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303655 5748 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:23:36.304224 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303658 5748 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:23:36.304224 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303661 5748 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:23:36.304224 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303664 5748 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:23:36.304224 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303667 5748 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:23:36.304224 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303670 5748 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:23:36.304662 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303672 5748 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:23:36.304662 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303675 5748 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:23:36.304662 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303678 5748 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:23:36.304662 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303681 5748 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:23:36.304662 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303684 5748 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:23:36.304662 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303688 5748 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:23:36.304662 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303693 5748 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:23:36.304662 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303698 5748 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:23:36.304662 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303701 5748 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:23:36.304662 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303705 5748 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:23:36.304662 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303708 5748 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:23:36.304662 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303711 5748 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:23:36.304662 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303714 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:23:36.304662 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303716 5748 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:23:36.304662 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303719 5748 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:23:36.304662 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303722 5748 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:23:36.304662 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303725 5748 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:23:36.304662 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303727 5748 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:23:36.304662 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303731 5748 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:23:36.305147 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303733 5748 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:23:36.305147 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303736 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:23:36.305147 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303738 5748 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:23:36.305147 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303741 5748 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:23:36.305147 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303745 5748 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:23:36.305147 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303747 5748 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:23:36.305147 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303750 5748 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:23:36.305147 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303753 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:23:36.305147 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303756 5748 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:23:36.305147 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303759 5748 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:23:36.305147 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303761 5748 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:23:36.305147 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303764 5748 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:23:36.305147 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303767 5748 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:23:36.305147 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303770 5748 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:23:36.305147 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303772 5748 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:23:36.305147 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303775 5748 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:23:36.305147 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303777 5748 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:23:36.305147 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303804 5748 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:23:36.305147 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303809 5748 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:23:36.305147 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303814 5748 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:23:36.305147 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303817 5748 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:23:36.305669 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303819 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:23:36.305669 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303822 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:23:36.305669 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303825 5748 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:23:36.305669 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303827 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:23:36.305669 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303830 5748 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:23:36.305669 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303833 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:23:36.305669 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303835 5748 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:23:36.305669 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303838 5748 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:23:36.305669 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303840 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:23:36.305669 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303843 5748 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:23:36.305669 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303846 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:23:36.305669 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303848 5748 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:23:36.305669 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303851 5748 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:23:36.305669 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303854 5748 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:23:36.305669 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303858 5748 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:23:36.305669 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303860 5748 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:23:36.305669 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303863 5748 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:23:36.305669 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303866 5748 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:23:36.305669 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303869 5748 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:23:36.306405 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303871 5748 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:23:36.306405 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303874 5748 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:23:36.306405 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303877 5748 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:23:36.306405 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.303879 5748 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:23:36.306405 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304709 5748 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:23:36.306405 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304717 5748 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:23:36.306405 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304721 5748 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:23:36.306405 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304724 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:23:36.306405 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304727 5748 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:23:36.306405 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304730 5748 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:23:36.306405 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304733 5748 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:23:36.306405 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304736 5748 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:23:36.306405 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304739 5748 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:23:36.306405 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304742 5748 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:23:36.306405 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304745 5748 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:23:36.306405 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304748 5748 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:23:36.306405 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304751 5748 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:23:36.306405 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304753 5748 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:23:36.306405 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304756 5748 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:23:36.307124 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304758 5748 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:23:36.307124 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304761 5748 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:23:36.307124 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304763 5748 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:23:36.307124 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304766 5748 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:23:36.307124 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304769 5748 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:23:36.307124 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304771 5748 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:23:36.307124 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304774 5748 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:23:36.307124 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304777 5748 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:23:36.307124 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304780 5748 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:23:36.307124 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304783 5748 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:23:36.307124 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304785 5748 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:23:36.307124 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304788 5748 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:23:36.307124 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304791 5748 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:23:36.307124 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304794 5748 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:23:36.307124 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304796 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:23:36.307124 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304799 5748 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:23:36.307124 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304801 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:23:36.307124 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304804 5748 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:23:36.307124 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304807 5748 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:23:36.307124 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304809 5748 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:23:36.307877 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304811 5748 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:23:36.307877 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304814 5748 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:23:36.307877 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304816 5748 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:23:36.307877 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304819 5748 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:23:36.307877 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304821 5748 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:23:36.307877 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304824 5748 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:23:36.307877 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304827 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:23:36.307877 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304832 5748 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:23:36.307877 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304836 5748 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:23:36.307877 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304839 5748 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:23:36.307877 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304841 5748 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:23:36.307877 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304845 5748 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:23:36.307877 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304848 5748 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:23:36.307877 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304851 5748 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:23:36.307877 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304854 5748 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:23:36.307877 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304857 5748 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:23:36.307877 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304859 5748 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:23:36.307877 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304862 5748 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:23:36.307877 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304864 5748 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:23:36.308678 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304867 5748 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:23:36.308678 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304870 5748 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:23:36.308678 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304872 5748 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:23:36.308678 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304874 5748 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:23:36.308678 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304877 5748 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:23:36.308678 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304880 5748 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:23:36.308678 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304882 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:23:36.308678 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304885 5748 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:23:36.308678 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304887 5748 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:23:36.308678 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304890 5748 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:23:36.308678 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304908 5748 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:23:36.308678 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304913 5748 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:23:36.308678 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304917 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:23:36.308678 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304922 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:23:36.308678 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304937 5748 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:23:36.308678 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304941 5748 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:23:36.308678 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304945 5748 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:23:36.308678 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304948 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:23:36.308678 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304951 5748 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:23:36.309490 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304953 5748 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:23:36.309490 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304957 5748 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:23:36.309490 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304959 5748 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:23:36.309490 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304962 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:23:36.309490 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304966 5748 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:23:36.309490 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304969 5748 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:23:36.309490 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304971 5748 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:23:36.309490 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304974 5748 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:23:36.309490 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304977 5748 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:23:36.309490 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304979 5748 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:23:36.309490 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304982 5748 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:23:36.309490 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304985 5748 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:23:36.309490 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304987 5748 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:23:36.309490 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304990 5748 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:23:36.309490 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304992 5748 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:23:36.309490 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304995 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:23:36.309490 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.304997 5748 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:23:36.309490 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305000 5748 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:23:36.309490 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305002 5748 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:23:36.309490 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305005 5748 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:23:36.310426 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305011 5748 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:23:36.310426 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305013 5748 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:23:36.310426 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305016 5748 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:23:36.310426 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305019 5748 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:23:36.310426 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305022 5748 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:23:36.310426 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305024 5748 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:23:36.310426 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305027 5748 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:23:36.310426 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305030 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:23:36.310426 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305033 5748 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:23:36.310426 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305036 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:23:36.310426 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305038 5748 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:23:36.310426 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305041 5748 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:23:36.310426 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305045 5748 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:23:36.310426 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305676 5748 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:23:36.310426 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305687 5748 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:23:36.310426 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305693 5748 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:23:36.310426 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305701 5748 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:23:36.310426 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305706 5748 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:23:36.310426 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305711 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:23:36.310426 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305716 5748 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:23:36.311421 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305720 5748 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:23:36.311421 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305724 5748 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:23:36.311421 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305728 5748 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:23:36.311421 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305732 5748 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:23:36.311421 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305736 5748 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:23:36.311421 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305740 5748 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:23:36.311421 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305744 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:23:36.311421 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305749 5748 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:23:36.311421 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305753 5748 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:23:36.311421 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305757 5748 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:23:36.311421 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305761 5748 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:23:36.311421 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305765 5748 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:23:36.311421 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305770 5748 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:23:36.311421 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305774 5748 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:23:36.311421 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305781 5748 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:23:36.311421 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305785 5748 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:23:36.311421 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305789 5748 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:23:36.311421 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305796 5748 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:23:36.311421 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305803 5748 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:23:36.312248 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305808 5748 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:23:36.312248 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305812 5748 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:23:36.312248 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305816 5748 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:23:36.312248 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305820 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:23:36.312248 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305824 5748 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:23:36.312248 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305828 5748 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:23:36.312248 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305832 5748 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:23:36.312248 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305837 5748 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:23:36.312248 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305842 5748 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:23:36.312248 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305846 5748 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:23:36.312248 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305850 5748 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:23:36.312248 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305854 5748 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:23:36.312248 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305860 5748 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:23:36.312248 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305864 5748 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:23:36.312248 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305868 5748 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:23:36.312248 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305872 5748 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:23:36.312248 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305877 5748 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:23:36.312248 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305881 5748 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:23:36.312248 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305885 5748 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:23:36.313058 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305889 5748 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:23:36.313058 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305908 5748 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:23:36.313058 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305913 5748 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:23:36.313058 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305917 5748 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:23:36.313058 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305921 5748 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:23:36.313058 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305926 5748 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:23:36.313058 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305930 5748 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:23:36.313058 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305937 5748 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:23:36.313058 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305945 5748 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:23:36.313058 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305950 5748 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:23:36.313058 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305955 5748 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:23:36.313058 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305959 5748 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:23:36.313058 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305964 5748 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:23:36.313058 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305969 5748 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:23:36.313058 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305973 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:23:36.313058 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305977 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:23:36.313058 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305981 5748 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:23:36.313058 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305986 5748 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:23:36.313058 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305990 5748 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:23:36.314069 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305994 5748 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:23:36.314069 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.305999 5748 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:23:36.314069 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.306003 5748 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:23:36.314069 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.306008 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:23:36.314069 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.306012 5748 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:23:36.314069 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.306016 5748 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:23:36.314069 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.306020 5748 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:23:36.314069 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.306024 5748 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:23:36.314069 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.306028 5748 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:23:36.314069 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.306033 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:23:36.314069 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.306037 5748 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:23:36.314069 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.306041 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:23:36.314069 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.306045 5748 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:23:36.314069 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.306048 5748 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:23:36.314069 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.306053 5748 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:23:36.314069 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.306057 5748 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:23:36.314069 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.306061 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:23:36.314069 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.306065 5748 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:23:36.314069 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.306069 5748 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:23:36.314795 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.306073 5748 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:23:36.314795 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.306077 5748 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:23:36.314795 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.306083 5748 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:23:36.314795 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.306087 5748 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:23:36.314795 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.306091 5748 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:23:36.314795 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.306096 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:23:36.314795 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.306100 5748 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:23:36.314795 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.306104 5748 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:23:36.314795 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.306108 5748 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:23:36.314795 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.306112 5748 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:23:36.314795 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.306116 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:23:36.314795 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.306120 5748 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:23:36.314795 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.306125 5748 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:23:36.314795 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.306129 5748 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:23:36.314795 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.306134 5748 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:23:36.314795 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.306138 5748 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:23:36.314795 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.306142 5748 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:23:36.314795 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.306146 5748 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:23:36.314795 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.306150 5748 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:23:36.314795 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.306154 5748 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:23:36.315543 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.306158 5748 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:23:36.315543 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.306162 5748 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:23:36.315543 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.306166 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:23:36.315543 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306306 5748 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 21:23:36.315543 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306321 5748 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 21:23:36.315543 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306330 5748 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 21:23:36.315543 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306337 5748 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 21:23:36.315543 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306344 5748 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 21:23:36.315543 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306349 5748 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 21:23:36.315543 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306356 5748 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 21:23:36.315543 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306363 5748 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 21:23:36.315543 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306368 5748 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 21:23:36.315543 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306373 5748 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 21:23:36.315543 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306378 5748 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 21:23:36.315543 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306385 5748 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 21:23:36.315543 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306390 5748 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 21:23:36.315543 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306394 5748 flags.go:64] FLAG: --cgroup-root="" Oct 30 21:23:36.315543 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306399 5748 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 21:23:36.315543 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306403 5748 flags.go:64] FLAG: --client-ca-file="" Oct 30 21:23:36.315543 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306408 5748 flags.go:64] FLAG: --cloud-provider="external" Oct 30 21:23:36.315543 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306412 5748 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 21:23:36.315543 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306418 5748 flags.go:64] FLAG: --cluster-domain="" Oct 30 21:23:36.315543 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306443 5748 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 21:23:36.316437 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306449 5748 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 21:23:36.316437 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306462 5748 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 21:23:36.316437 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306468 5748 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 21:23:36.316437 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306474 5748 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 21:23:36.316437 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306479 5748 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 21:23:36.316437 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306484 5748 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 21:23:36.316437 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306490 5748 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 21:23:36.316437 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306495 5748 flags.go:64] FLAG: --contention-profiling="false" Oct 30 21:23:36.316437 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306499 5748 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 21:23:36.316437 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306504 5748 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 21:23:36.316437 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306509 5748 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 21:23:36.316437 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306513 5748 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 21:23:36.316437 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306520 5748 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 21:23:36.316437 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306524 5748 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 21:23:36.316437 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306529 5748 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 21:23:36.316437 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306538 5748 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 21:23:36.316437 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306545 5748 flags.go:64] FLAG: --enable-server="true" Oct 30 21:23:36.316437 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306565 5748 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 21:23:36.316437 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306574 5748 flags.go:64] FLAG: --event-burst="100" Oct 30 21:23:36.316437 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306579 5748 flags.go:64] FLAG: --event-qps="50" Oct 30 21:23:36.316437 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306583 5748 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 21:23:36.316437 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306599 5748 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 21:23:36.316437 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306603 5748 flags.go:64] FLAG: --eviction-hard="" Oct 30 21:23:36.316437 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306612 5748 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 21:23:36.317212 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306616 5748 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 21:23:36.317212 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306621 5748 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 21:23:36.317212 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306626 5748 flags.go:64] FLAG: --eviction-soft="" Oct 30 21:23:36.317212 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306630 5748 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 21:23:36.317212 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306635 5748 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 21:23:36.317212 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306640 5748 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 21:23:36.317212 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306645 5748 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 21:23:36.317212 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306649 5748 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 21:23:36.317212 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306653 5748 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 21:23:36.317212 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306657 5748 flags.go:64] FLAG: --feature-gates="" Oct 30 21:23:36.317212 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306663 5748 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 21:23:36.317212 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306668 5748 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 21:23:36.317212 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306673 5748 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 21:23:36.317212 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306677 5748 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 21:23:36.317212 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306682 5748 flags.go:64] FLAG: --healthz-port="10248" Oct 30 21:23:36.317212 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306687 5748 flags.go:64] FLAG: --help="false" Oct 30 21:23:36.317212 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306692 5748 flags.go:64] FLAG: --hostname-override="ip-10-0-143-61.us-west-1.compute.internal" Oct 30 21:23:36.317212 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306696 5748 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 21:23:36.317212 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306700 5748 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 21:23:36.317212 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306705 5748 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 21:23:36.317212 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306710 5748 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 21:23:36.317212 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306715 5748 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 21:23:36.317212 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306719 5748 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 21:23:36.317826 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306724 5748 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 21:23:36.317826 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306730 5748 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 21:23:36.317826 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306734 5748 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 21:23:36.317826 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306739 5748 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 21:23:36.317826 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306744 5748 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 21:23:36.317826 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306748 5748 flags.go:64] FLAG: --kube-reserved="" Oct 30 21:23:36.317826 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306753 5748 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 21:23:36.317826 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306757 5748 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 21:23:36.317826 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306763 5748 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 21:23:36.317826 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306768 5748 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 21:23:36.317826 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306772 5748 flags.go:64] FLAG: --lock-file="" Oct 30 21:23:36.317826 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306776 5748 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 21:23:36.317826 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306781 5748 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 21:23:36.317826 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306786 5748 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 21:23:36.317826 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306794 5748 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 21:23:36.317826 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306799 5748 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 21:23:36.317826 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306804 5748 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 21:23:36.317826 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306808 5748 flags.go:64] FLAG: --logging-format="text" Oct 30 21:23:36.317826 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306813 5748 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 21:23:36.317826 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306818 5748 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 21:23:36.317826 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306823 5748 flags.go:64] FLAG: --manifest-url="" Oct 30 21:23:36.317826 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306828 5748 flags.go:64] FLAG: --manifest-url-header="" Oct 30 21:23:36.317826 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306835 5748 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 21:23:36.317826 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306839 5748 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 21:23:36.317826 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306845 5748 flags.go:64] FLAG: --max-pods="110" Oct 30 21:23:36.318508 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306851 5748 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 21:23:36.318508 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306855 5748 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 21:23:36.318508 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306859 5748 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 21:23:36.318508 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306864 5748 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 21:23:36.318508 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306868 5748 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 21:23:36.318508 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306873 5748 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 21:23:36.318508 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306877 5748 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 21:23:36.318508 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306888 5748 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 21:23:36.318508 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306908 5748 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 21:23:36.318508 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306914 5748 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 21:23:36.318508 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306919 5748 flags.go:64] FLAG: --pod-cidr="" Oct 30 21:23:36.318508 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306923 5748 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 21:23:36.318508 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306927 5748 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 21:23:36.318508 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306932 5748 flags.go:64] FLAG: --pods-per-core="0" Oct 30 21:23:36.318508 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306936 5748 flags.go:64] FLAG: --port="10250" Oct 30 21:23:36.318508 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306942 5748 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 21:23:36.318508 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306947 5748 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0705c1c65e331f110" Oct 30 21:23:36.318508 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306952 5748 flags.go:64] FLAG: --qos-reserved="" Oct 30 21:23:36.318508 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306956 5748 flags.go:64] FLAG: --read-only-port="10255" Oct 30 21:23:36.318508 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306961 5748 flags.go:64] FLAG: --register-node="true" Oct 30 21:23:36.318508 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306965 5748 flags.go:64] FLAG: --register-with-taints="" Oct 30 21:23:36.318508 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306971 5748 flags.go:64] FLAG: --registry-burst="10" Oct 30 21:23:36.318508 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306976 5748 flags.go:64] FLAG: --registry-qps="5" Oct 30 21:23:36.318508 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306980 5748 flags.go:64] FLAG: --reserved-cpus="" Oct 30 21:23:36.318508 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306985 5748 flags.go:64] FLAG: --reserved-memory="" Oct 30 21:23:36.318508 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306991 5748 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 21:23:36.319153 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.306995 5748 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 21:23:36.319153 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.307000 5748 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 21:23:36.319153 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.307005 5748 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 21:23:36.319153 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.307009 5748 flags.go:64] FLAG: --runonce="false" Oct 30 21:23:36.319153 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.307014 5748 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 21:23:36.319153 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.307019 5748 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 21:23:36.319153 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.307024 5748 flags.go:64] FLAG: --seccomp-default="false" Oct 30 21:23:36.319153 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.307029 5748 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 21:23:36.319153 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.307034 5748 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 21:23:36.319153 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.307038 5748 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 21:23:36.319153 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.307043 5748 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 21:23:36.319153 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.307048 5748 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 21:23:36.319153 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.307052 5748 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 21:23:36.319153 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.307057 5748 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 21:23:36.319153 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.307063 5748 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 21:23:36.319153 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.307067 5748 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 21:23:36.319153 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.307073 5748 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 21:23:36.319153 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.307078 5748 flags.go:64] FLAG: --system-cgroups="" Oct 30 21:23:36.319153 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.307082 5748 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 21:23:36.319153 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.307091 5748 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 21:23:36.319153 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.307096 5748 flags.go:64] FLAG: --tls-cert-file="" Oct 30 21:23:36.319153 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.307102 5748 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 21:23:36.319153 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.307109 5748 flags.go:64] FLAG: --tls-min-version="" Oct 30 21:23:36.319153 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.307113 5748 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 21:23:36.319153 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.307118 5748 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 21:23:36.320033 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.307122 5748 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 21:23:36.320033 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.307127 5748 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 21:23:36.320033 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.307132 5748 flags.go:64] FLAG: --v="2" Oct 30 21:23:36.320033 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.307139 5748 flags.go:64] FLAG: --version="false" Oct 30 21:23:36.320033 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.307145 5748 flags.go:64] FLAG: --vmodule="" Oct 30 21:23:36.320033 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.307152 5748 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 21:23:36.320033 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.307158 5748 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 21:23:36.320033 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307376 5748 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:23:36.320033 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307383 5748 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:23:36.320033 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307388 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:23:36.320033 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307393 5748 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:23:36.320033 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307397 5748 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:23:36.320033 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307401 5748 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:23:36.320033 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307406 5748 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:23:36.320033 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307410 5748 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:23:36.320033 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307415 5748 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:23:36.320033 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307420 5748 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:23:36.320033 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307424 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:23:36.320033 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307428 5748 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:23:36.320033 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307433 5748 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:23:36.320922 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307437 5748 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:23:36.320922 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307443 5748 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:23:36.320922 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307447 5748 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:23:36.320922 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307451 5748 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:23:36.320922 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307455 5748 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:23:36.320922 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307459 5748 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:23:36.320922 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307464 5748 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:23:36.320922 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307468 5748 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:23:36.320922 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307473 5748 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:23:36.320922 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307478 5748 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:23:36.320922 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307482 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:23:36.320922 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307486 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:23:36.320922 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307490 5748 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:23:36.320922 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307494 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:23:36.320922 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307498 5748 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:23:36.320922 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307502 5748 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:23:36.320922 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307506 5748 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:23:36.320922 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307510 5748 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:23:36.320922 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307514 5748 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:23:36.320922 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307518 5748 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:23:36.321628 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307522 5748 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:23:36.321628 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307526 5748 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:23:36.321628 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307529 5748 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:23:36.321628 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307534 5748 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:23:36.321628 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307538 5748 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:23:36.321628 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307541 5748 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:23:36.321628 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307545 5748 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:23:36.321628 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307549 5748 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:23:36.321628 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307554 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:23:36.321628 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307558 5748 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:23:36.321628 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307562 5748 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:23:36.321628 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307566 5748 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:23:36.321628 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307569 5748 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:23:36.321628 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307575 5748 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:23:36.321628 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307579 5748 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:23:36.321628 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307583 5748 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:23:36.321628 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307587 5748 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:23:36.321628 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307591 5748 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:23:36.321628 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307596 5748 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:23:36.321628 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307600 5748 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:23:36.322255 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307604 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:23:36.322255 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307608 5748 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:23:36.322255 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307613 5748 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:23:36.322255 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307617 5748 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:23:36.322255 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307621 5748 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:23:36.322255 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307625 5748 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:23:36.322255 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307629 5748 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:23:36.322255 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307633 5748 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:23:36.322255 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307637 5748 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:23:36.322255 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307641 5748 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:23:36.322255 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307646 5748 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:23:36.322255 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307650 5748 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:23:36.322255 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307654 5748 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:23:36.322255 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307658 5748 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:23:36.322255 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307662 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:23:36.322255 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307666 5748 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:23:36.322255 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307671 5748 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:23:36.322255 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307675 5748 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:23:36.322255 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307679 5748 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:23:36.322255 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307683 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:23:36.322840 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307687 5748 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:23:36.322840 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307693 5748 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:23:36.322840 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307698 5748 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:23:36.322840 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307702 5748 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:23:36.322840 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307705 5748 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:23:36.322840 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307711 5748 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:23:36.322840 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307715 5748 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:23:36.322840 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307720 5748 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:23:36.322840 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307724 5748 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:23:36.322840 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307728 5748 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:23:36.322840 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307732 5748 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:23:36.322840 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307736 5748 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:23:36.322840 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307740 5748 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:23:36.322840 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307744 5748 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:23:36.322840 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307748 5748 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:23:36.322840 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307753 5748 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:23:36.322840 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307757 5748 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:23:36.322840 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307760 5748 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:23:36.322840 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307764 5748 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:23:36.322840 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307772 5748 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:23:36.323527 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307776 5748 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:23:36.323527 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307780 5748 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:23:36.323527 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307784 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:23:36.323527 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307790 5748 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:23:36.323527 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307795 5748 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:23:36.323527 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307801 5748 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:23:36.323527 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307805 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:23:36.323527 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307810 5748 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:23:36.323527 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307816 5748 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:23:36.323527 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307822 5748 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:23:36.323527 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307827 5748 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:23:36.323527 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307832 5748 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:23:36.323527 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307836 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:23:36.323527 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.307889 5748 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:23:36.323527 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307964 5748 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:23:36.324134 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307970 5748 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:23:36.324134 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307975 5748 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:23:36.324134 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307982 5748 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:23:36.324134 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307987 5748 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:23:36.324134 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307990 5748 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:23:36.324134 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307994 5748 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:23:36.324134 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.307998 5748 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:23:36.324134 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308002 5748 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:23:36.324134 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308006 5748 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:23:36.324134 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308010 5748 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:23:36.324134 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308015 5748 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:23:36.324134 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308019 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:23:36.324134 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308023 5748 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:23:36.324134 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308029 5748 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:23:36.324134 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308033 5748 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:23:36.324134 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308038 5748 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:23:36.324134 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308042 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:23:36.324134 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308046 5748 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:23:36.324134 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308050 5748 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:23:36.324134 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308054 5748 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:23:36.324861 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308058 5748 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:23:36.324861 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308062 5748 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:23:36.324861 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308067 5748 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:23:36.324861 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308071 5748 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:23:36.324861 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308075 5748 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:23:36.324861 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308079 5748 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:23:36.324861 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308084 5748 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:23:36.324861 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308088 5748 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:23:36.324861 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308092 5748 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:23:36.324861 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308096 5748 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:23:36.324861 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308100 5748 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:23:36.324861 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308105 5748 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:23:36.324861 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308110 5748 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:23:36.324861 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308114 5748 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:23:36.324861 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308123 5748 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:23:36.324861 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308126 5748 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:23:36.324861 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308130 5748 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:23:36.324861 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308135 5748 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:23:36.324861 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308145 5748 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:23:36.324861 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308149 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:23:36.325671 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308155 5748 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:23:36.325671 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308162 5748 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:23:36.325671 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308166 5748 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:23:36.325671 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308170 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:23:36.325671 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308175 5748 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:23:36.325671 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308183 5748 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:23:36.325671 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308188 5748 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:23:36.325671 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308193 5748 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:23:36.325671 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308198 5748 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:23:36.325671 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308203 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:23:36.325671 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308207 5748 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:23:36.325671 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308211 5748 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:23:36.325671 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308215 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:23:36.325671 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308220 5748 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:23:36.325671 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308224 5748 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:23:36.325671 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308228 5748 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:23:36.325671 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308233 5748 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:23:36.325671 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308238 5748 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:23:36.326468 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308242 5748 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:23:36.326468 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308246 5748 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:23:36.326468 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308251 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:23:36.326468 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308255 5748 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:23:36.326468 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308259 5748 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:23:36.326468 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308263 5748 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:23:36.326468 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308268 5748 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:23:36.326468 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308273 5748 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:23:36.326468 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308279 5748 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:23:36.326468 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308283 5748 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:23:36.326468 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308287 5748 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:23:36.326468 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308292 5748 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:23:36.326468 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308296 5748 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:23:36.326468 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308300 5748 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:23:36.326468 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308305 5748 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:23:36.326468 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308309 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:23:36.326468 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308314 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:23:36.326468 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308320 5748 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:23:36.326468 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308324 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:23:36.327175 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308330 5748 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:23:36.327175 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308334 5748 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:23:36.327175 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308338 5748 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:23:36.327175 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308342 5748 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:23:36.327175 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308346 5748 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:23:36.327175 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308351 5748 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:23:36.327175 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308355 5748 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:23:36.327175 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308360 5748 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:23:36.327175 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308364 5748 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:23:36.327175 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308368 5748 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:23:36.327175 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308372 5748 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:23:36.327175 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308376 5748 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:23:36.327175 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308380 5748 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:23:36.327175 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308385 5748 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:23:36.327175 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308388 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:23:36.327175 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308392 5748 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:23:36.327175 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308396 5748 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:23:36.327175 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308400 5748 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:23:36.327175 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308405 5748 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:23:36.327175 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308409 5748 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:23:36.327822 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308413 5748 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:23:36.327822 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308421 5748 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:23:36.327822 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308425 5748 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:23:36.327822 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308429 5748 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:23:36.327822 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308433 5748 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:23:36.327822 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308437 5748 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:23:36.327822 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308442 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:23:36.327822 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.308446 5748 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:23:36.327822 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.312935 5748 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 21:23:36.327822 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.312959 5748 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 21:23:36.327822 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.312979 5748 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 21:23:36.327822 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.312984 5748 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 21:23:36.327822 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313051 5748 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:23:36.327822 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313057 5748 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:23:36.327822 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313060 5748 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:23:36.327822 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313063 5748 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:23:36.327822 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313066 5748 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:23:36.327822 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313068 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:23:36.327822 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313071 5748 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:23:36.327822 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313075 5748 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:23:36.328342 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313078 5748 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:23:36.328342 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313081 5748 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:23:36.328342 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313084 5748 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:23:36.328342 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313086 5748 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:23:36.328342 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313089 5748 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:23:36.328342 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313093 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:23:36.328342 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313097 5748 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:23:36.328342 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313101 5748 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:23:36.328342 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313104 5748 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:23:36.328342 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313107 5748 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:23:36.328342 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313110 5748 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:23:36.328342 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313113 5748 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:23:36.328342 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313115 5748 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:23:36.328342 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313118 5748 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:23:36.328342 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313121 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:23:36.328342 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313124 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:23:36.328342 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313127 5748 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:23:36.328342 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313130 5748 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:23:36.328342 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313133 5748 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:23:36.328795 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313136 5748 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:23:36.328795 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313138 5748 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:23:36.328795 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313149 5748 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:23:36.328795 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313152 5748 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:23:36.328795 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313155 5748 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:23:36.328795 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313157 5748 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:23:36.328795 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313160 5748 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:23:36.328795 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313163 5748 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:23:36.328795 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313166 5748 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:23:36.328795 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313168 5748 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:23:36.328795 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313171 5748 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:23:36.328795 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313174 5748 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:23:36.328795 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313176 5748 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:23:36.328795 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313179 5748 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:23:36.328795 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313182 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:23:36.328795 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313184 5748 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:23:36.328795 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313187 5748 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:23:36.328795 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313190 5748 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:23:36.328795 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313193 5748 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:23:36.328795 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313196 5748 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:23:36.329385 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313198 5748 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:23:36.329385 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313201 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:23:36.329385 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313204 5748 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:23:36.329385 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313206 5748 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:23:36.329385 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313209 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:23:36.329385 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313212 5748 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:23:36.329385 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313214 5748 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:23:36.329385 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313217 5748 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:23:36.329385 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313220 5748 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:23:36.329385 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313222 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:23:36.329385 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313225 5748 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:23:36.329385 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313227 5748 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:23:36.329385 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313230 5748 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:23:36.329385 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313232 5748 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:23:36.329385 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313235 5748 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:23:36.329385 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313243 5748 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:23:36.329385 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313246 5748 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:23:36.329385 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313249 5748 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:23:36.329385 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313251 5748 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:23:36.330167 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313254 5748 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:23:36.330167 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313256 5748 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:23:36.330167 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313259 5748 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:23:36.330167 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313262 5748 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:23:36.330167 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313264 5748 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:23:36.330167 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313267 5748 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:23:36.330167 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313271 5748 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:23:36.330167 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313275 5748 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:23:36.330167 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313279 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:23:36.330167 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313283 5748 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:23:36.330167 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313286 5748 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:23:36.330167 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313289 5748 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:23:36.330167 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313292 5748 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:23:36.330167 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313294 5748 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:23:36.330167 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313297 5748 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:23:36.330167 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313300 5748 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:23:36.330167 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313302 5748 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:23:36.330167 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313305 5748 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:23:36.330167 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313307 5748 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:23:36.330979 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313310 5748 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:23:36.330979 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313313 5748 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:23:36.330979 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313315 5748 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:23:36.330979 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313318 5748 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:23:36.330979 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313320 5748 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:23:36.330979 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313323 5748 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:23:36.330979 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313326 5748 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:23:36.330979 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313328 5748 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:23:36.330979 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313331 5748 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:23:36.330979 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313334 5748 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:23:36.330979 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313342 5748 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:23:36.330979 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313344 5748 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:23:36.330979 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313347 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:23:36.330979 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313350 5748 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:23:36.330979 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313352 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:23:36.330979 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313355 5748 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:23:36.330979 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313358 5748 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:23:36.330979 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313361 5748 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:23:36.330979 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313364 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:23:36.330979 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313366 5748 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:23:36.331801 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313369 5748 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:23:36.331801 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.313428 5748 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:23:36.331801 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313670 5748 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:23:36.331801 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313679 5748 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:23:36.331801 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313683 5748 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:23:36.331801 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313689 5748 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:23:36.331801 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313694 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:23:36.331801 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313708 5748 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:23:36.331801 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313713 5748 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:23:36.331801 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313717 5748 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:23:36.331801 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313721 5748 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:23:36.331801 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313726 5748 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:23:36.331801 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313730 5748 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:23:36.331801 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313734 5748 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:23:36.331801 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313740 5748 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:23:36.331801 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313744 5748 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:23:36.332307 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313749 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:23:36.332307 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313753 5748 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:23:36.332307 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313757 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:23:36.332307 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313767 5748 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:23:36.332307 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313772 5748 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:23:36.332307 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313776 5748 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:23:36.332307 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313781 5748 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:23:36.332307 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313799 5748 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:23:36.332307 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313803 5748 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:23:36.332307 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313830 5748 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:23:36.332307 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313837 5748 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:23:36.332307 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313842 5748 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:23:36.332307 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313847 5748 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:23:36.332307 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313851 5748 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:23:36.332307 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313856 5748 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:23:36.332307 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313860 5748 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:23:36.332307 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313866 5748 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:23:36.332307 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313870 5748 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:23:36.332307 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313874 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:23:36.332849 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313883 5748 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:23:36.332849 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313887 5748 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:23:36.332849 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313892 5748 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:23:36.332849 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313912 5748 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:23:36.332849 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313917 5748 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:23:36.332849 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313922 5748 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:23:36.332849 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313926 5748 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:23:36.332849 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313930 5748 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:23:36.332849 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313934 5748 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:23:36.332849 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313938 5748 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:23:36.332849 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313942 5748 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:23:36.332849 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313947 5748 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:23:36.332849 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313951 5748 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:23:36.332849 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313955 5748 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:23:36.332849 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313960 5748 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:23:36.332849 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313964 5748 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:23:36.332849 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313968 5748 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:23:36.332849 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313972 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:23:36.332849 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313976 5748 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:23:36.332849 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313980 5748 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:23:36.333445 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.313986 5748 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:23:36.333445 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314001 5748 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:23:36.333445 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314005 5748 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:23:36.333445 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314009 5748 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:23:36.333445 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314013 5748 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:23:36.333445 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314016 5748 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:23:36.333445 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314020 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:23:36.333445 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314024 5748 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:23:36.333445 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314029 5748 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:23:36.333445 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314033 5748 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:23:36.333445 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314037 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:23:36.333445 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314041 5748 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:23:36.333445 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314045 5748 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:23:36.333445 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314050 5748 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:23:36.333445 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314054 5748 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:23:36.333445 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314058 5748 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:23:36.333445 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314062 5748 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:23:36.333445 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314065 5748 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:23:36.333445 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314069 5748 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:23:36.333910 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314073 5748 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:23:36.333910 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314077 5748 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:23:36.333910 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314080 5748 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:23:36.333910 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314084 5748 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:23:36.333910 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314095 5748 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:23:36.333910 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314099 5748 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:23:36.333910 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314104 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:23:36.333910 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314109 5748 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:23:36.333910 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314113 5748 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:23:36.333910 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314117 5748 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:23:36.333910 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314121 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:23:36.333910 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314125 5748 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:23:36.333910 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314129 5748 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:23:36.333910 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314137 5748 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:23:36.333910 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314142 5748 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:23:36.333910 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314154 5748 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:23:36.333910 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314158 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:23:36.333910 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314162 5748 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:23:36.334371 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314166 5748 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:23:36.334371 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314169 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:23:36.334371 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314174 5748 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:23:36.334371 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314178 5748 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:23:36.334371 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314183 5748 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:23:36.334371 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314187 5748 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:23:36.334371 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314190 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:23:36.334371 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314194 5748 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:23:36.334371 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314198 5748 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:23:36.334371 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314203 5748 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:23:36.334371 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314207 5748 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:23:36.334371 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314211 5748 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:23:36.334371 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314215 5748 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:23:36.334371 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314219 5748 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:23:36.334371 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314223 5748 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:23:36.334371 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314227 5748 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:23:36.334371 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.314280 5748 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:23:36.334790 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314341 5748 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:23:36.334790 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314348 5748 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:23:36.334790 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314352 5748 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:23:36.334790 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314356 5748 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:23:36.334790 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314361 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:23:36.334790 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314365 5748 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:23:36.334790 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314370 5748 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:23:36.334790 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314374 5748 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:23:36.334790 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314378 5748 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:23:36.334790 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314403 5748 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:23:36.334790 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314409 5748 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:23:36.334790 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314416 5748 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:23:36.334790 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314420 5748 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:23:36.334790 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314431 5748 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:23:36.334790 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314435 5748 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:23:36.334790 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314439 5748 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:23:36.334790 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314443 5748 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:23:36.334790 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314448 5748 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:23:36.334790 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314452 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:23:36.334790 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314463 5748 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:23:36.334790 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314468 5748 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:23:36.335316 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314472 5748 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:23:36.335316 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314476 5748 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:23:36.335316 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314480 5748 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:23:36.335316 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314484 5748 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:23:36.335316 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314488 5748 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:23:36.335316 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314492 5748 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:23:36.335316 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314496 5748 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:23:36.335316 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314500 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:23:36.335316 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314504 5748 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:23:36.335316 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314507 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:23:36.335316 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314512 5748 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:23:36.335316 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314518 5748 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:23:36.335316 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314522 5748 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:23:36.335316 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314526 5748 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:23:36.335316 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314530 5748 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:23:36.335316 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314535 5748 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:23:36.335316 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314541 5748 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:23:36.335316 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314548 5748 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:23:36.335316 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314553 5748 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:23:36.335801 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314559 5748 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:23:36.335801 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314563 5748 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:23:36.335801 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314567 5748 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:23:36.335801 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314572 5748 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:23:36.335801 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314576 5748 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:23:36.335801 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314580 5748 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:23:36.335801 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314591 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:23:36.335801 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314596 5748 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:23:36.335801 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314600 5748 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:23:36.335801 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314605 5748 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:23:36.335801 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314609 5748 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:23:36.335801 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314615 5748 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:23:36.335801 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314619 5748 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:23:36.335801 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314623 5748 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:23:36.335801 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314628 5748 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:23:36.335801 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314632 5748 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:23:36.335801 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314636 5748 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:23:36.335801 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314640 5748 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:23:36.335801 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314644 5748 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:23:36.335801 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314648 5748 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:23:36.336294 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314651 5748 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:23:36.336294 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314655 5748 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:23:36.336294 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314659 5748 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:23:36.336294 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314663 5748 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:23:36.336294 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314670 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:23:36.336294 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314674 5748 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:23:36.336294 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314678 5748 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:23:36.336294 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314683 5748 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:23:36.336294 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314687 5748 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:23:36.336294 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314691 5748 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:23:36.336294 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314695 5748 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:23:36.336294 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314700 5748 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:23:36.336294 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314704 5748 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:23:36.336294 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314709 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:23:36.336294 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314713 5748 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:23:36.336294 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314718 5748 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:23:36.336294 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314722 5748 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:23:36.336294 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314727 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:23:36.336294 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314731 5748 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:23:36.336294 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314745 5748 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:23:36.336778 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314749 5748 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:23:36.336778 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314753 5748 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:23:36.336778 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314757 5748 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:23:36.336778 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314762 5748 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:23:36.336778 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314766 5748 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:23:36.336778 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314771 5748 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:23:36.336778 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314775 5748 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:23:36.336778 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314779 5748 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:23:36.336778 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314784 5748 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:23:36.336778 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314788 5748 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:23:36.336778 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314792 5748 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:23:36.336778 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314796 5748 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:23:36.336778 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314801 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:23:36.336778 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314806 5748 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:23:36.336778 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314810 5748 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:23:36.336778 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314814 5748 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:23:36.336778 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314820 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:23:36.336778 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314825 5748 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:23:36.336778 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314829 5748 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:23:36.337250 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314835 5748 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:23:36.337250 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314841 5748 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:23:36.337250 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314846 5748 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:23:36.337250 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314850 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:23:36.337250 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314857 5748 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:23:36.337250 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314861 5748 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:23:36.337250 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:36.314866 5748 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:23:36.337250 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.314988 5748 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 21:23:36.337250 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.316583 5748 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 21:23:36.337250 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.316705 5748 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 21:23:36.337250 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.317068 5748 server.go:1014] "Starting client certificate rotation" Oct 30 21:23:36.337250 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.317208 5748 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:23:36.337250 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.317320 5748 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:17:20 +0000 UTC" deadline="2028-04-19 20:34:29.199219035 +0000 UTC" Oct 30 21:23:36.337250 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.317384 5748 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12887h10m52.881837176s" Oct 30 21:23:36.337250 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.317992 5748 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:23:36.337687 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.320705 5748 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:23:36.337687 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.322651 5748 log.go:25] "Validated CRI v1 runtime API" Oct 30 21:23:36.337687 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.325104 5748 log.go:25] "Validated CRI v1 image API" Oct 30 21:23:36.337687 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.325978 5748 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 21:23:36.337687 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.328537 5748 fs.go:135] Filesystem UUIDs: map[509c864e-1681-4cb1-a673-5aa85d4d34b0:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 f2d395fa-97f1-4864-b1e7-a627def2f004:/dev/nvme0n1p4] Oct 30 21:23:36.337957 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.328564 5748 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/16851feff42b6b44a4b4f281e1734b22181590f88d68cd1bb7f73d0622bd9703/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/16851feff42b6b44a4b4f281e1734b22181590f88d68cd1bb7f73d0622bd9703/userdata/shm major:0 minor:251 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/21dac7bbe759d29d25155018511b5532b1b78d6e336d26183d4d7505e28570ab/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/21dac7bbe759d29d25155018511b5532b1b78d6e336d26183d4d7505e28570ab/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/253902cf7fb7a07a5987fe209537eb45f982a10286b6a15f35eacb7027da2cd8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/253902cf7fb7a07a5987fe209537eb45f982a10286b6a15f35eacb7027da2cd8/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/26cd448adaaa27c53605b8c3447aadf82556ccb34651e6e39acc6b4688aaf96a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/26cd448adaaa27c53605b8c3447aadf82556ccb34651e6e39acc6b4688aaf96a/userdata/shm major:0 minor:400 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2a4f46994282538de689d63e23d77357bf6940b5effeecfdd9ee73ecf275991b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2a4f46994282538de689d63e23d77357bf6940b5effeecfdd9ee73ecf275991b/userdata/shm major:0 minor:362 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2a6c6abcf5dc8b26d1d6d6f868f70a22fb169f9647327bab20ca6c3646432656/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2a6c6abcf5dc8b26d1d6d6f868f70a22fb169f9647327bab20ca6c3646432656/userdata/shm major:0 minor:276 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2e8ccef945c9b2ab26da19f76a836a73756872b7554e0a67985accacdcb37fe9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2e8ccef945c9b2ab26da19f76a836a73756872b7554e0a67985accacdcb37fe9/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/313b3e77bdc93435cc744be37f449318264b5b426dbe34048426a475aeadafcb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/313b3e77bdc93435cc744be37f449318264b5b426dbe34048426a475aeadafcb/userdata/shm major:0 minor:287 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/381f7b82d8dc76d1ea1c32732ba49780f515bf1815ff83c89f5870338b25d09c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/381f7b82d8dc76d1ea1c32732ba49780f515bf1815ff83c89f5870338b25d09c/userdata/shm major:0 minor:310 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6b2a31073c7737830fb55f6fe12435407df6541f64cf11219ac235d3ab7db600/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6b2a31073c7737830fb55f6fe12435407df6541f64cf11219ac235d3ab7db600/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6d53dc7170d4956d3c83148a21dfb0a03609822ffe6b368653e1331804f3df7d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6d53dc7170d4956d3c83148a21dfb0a03609822ffe6b368653e1331804f3df7d/userdata/shm major:0 minor:305 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7b1cb4f6a870992a27f8a7c72f56ffe89d0e2b5519d6fcad4ff407278af0bc83/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7b1cb4f6a870992a27f8a7c72f56ffe89d0e2b5519d6fcad4ff407278af0bc83/userdata/shm major:0 minor:291 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/839bfab96e7f8e0962d2a288948ea4e459936b713d5626d73efb3434d0f2d12a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/839bfab96e7f8e0962d2a288948ea4e459936b713d5626d73efb3434d0f2d12a/userdata/shm major:0 minor:312 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/90b5c7445469629806ce6590748554079f89b212ed99dfc9f33c2f39aea5951c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/90b5c7445469629806ce6590748554079f89b212ed99dfc9f33c2f39aea5951c/userdata/shm major:0 minor:430 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/953276e6631de9919ab79f85e9cad674e90cb10dc00b94a70fea3bf5cef97545/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/953276e6631de9919ab79f85e9cad674e90cb10dc00b94a70fea3bf5cef97545/userdata/shm major:0 minor:95 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9dc54f963eddd9f65ef3f9797b5563e449fed9af9fc472616366c4b6d508a75e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9dc54f963eddd9f65ef3f9797b5563e449fed9af9fc472616366c4b6d508a75e/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a283ea7030cc46493344d2445e51bba7633bc33dd6cd07a5f47a6bdef80a718b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a283ea7030cc46493344d2445e51bba7633bc33dd6cd07a5f47a6bdef80a718b/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ac3b2f70532aadcd84f622494b52343b96ea6fa8e52d777db57d766979f199fb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ac3b2f70532aadcd84f622494b52343b96ea6fa8e52d777db57d766979f199fb/userdata/shm major:0 minor:339 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b092a4deb6fd47f6577074ba533fcd9831761318517be661b91db0c2191c919f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b092a4deb6fd47f6577074ba533fcd9831761318517be661b91db0c2191c919f/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b4c99a19e2a768b0995b24beda1a0329650dd8b1feed61600397baf6c1a0abe2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b4c99a19e2a768b0995b24beda1a0329650dd8b1feed61600397baf6c1a0abe2/userdata/shm major:0 minor:264 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b594b27c3c927b7ab46df23306276cbd54084dabd3bf99c51864a383dc490c7d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b594b27c3c927b7ab46df23306276cbd54084dabd3bf99c51864a383dc490c7d/userdata/shm major:0 minor:289 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/be8bf0429e5f966f001d3467b942c1f2c894b8e25c67bdba0df0bce3b28e9b59/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/be8bf0429e5f966f001d3467b942c1f2c894b8e25c67bdba0df0bce3b28e9b59/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d5ffffe05af64a1646fb52a59c06c8823e1fb52b56b17d704bf19046a036f3b2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d5ffffe05af64a1646fb52a59c06c8823e1fb52b56b17d704bf19046a036f3b2/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e1b78fb648faa00672c441f6ea61fae944f1bcd635af480835decf0a41cc8f60/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e1b78fb648faa00672c441f6ea61fae944f1bcd635af480835decf0a41cc8f60/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ed963bfa9bbb231b5afbfba50c8b3914a24934b413908bda4c6f037527000a01/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ed963bfa9bbb231b5afbfba50c8b3914a24934b413908bda4c6f037527000a01/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ee9f13f99dd386c9fdec54d4ce3a925258a068ea279cbfbc01ee5a7d8534a773/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ee9f13f99dd386c9fdec54d4ce3a925258a068ea279cbfbc01ee5a7d8534a773/userdata/shm major:0 minor:423 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f6a7fe29d8810324e1e8f160b24c936f3766ae93c0e454b9d78eef9f84b05a86/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f6a7fe29d8810324e1e8f160b24c936f3766ae93c0e454b9d78eef9f84b05a86/userdata/shm major:0 minor:406 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ff15d90480ca95d335939e1f2e01526ffd17ff5dc66f00301b363e9c71972421/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ff15d90480ca95d335939e1f2e01526ffd17ff5dc66f00301b363e9c71972421/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/166e5f35-0987-4eef-917a-a4772acba82f/volumes/kubernetes.io~projected/kube-api-access-459jm:{mountpoint:/var/lib/kubelet/pods/166e5f35-0987-4eef-917a-a4772acba82f/volumes/kubernetes.io~projected/kube-api-access-459jm major:0 minor:304 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/166e5f35-0987-4eef-917a-a4772acba82f/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/166e5f35-0987-4eef-917a-a4772acba82f/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:302 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/166e5f35-0987-4eef-917a-a4772acba82f/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/166e5f35-0987-4eef-917a-a4772acba82f/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:303 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979/volumes/kubernetes.io~projected/kube-api-access-hs2w6:{mountpoint:/var/lib/kubelet/pods/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979/volumes/kubernetes.io~projected/kube-api-access-hs2w6 major:0 minor:309 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:307 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:308 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2496f073-7bf0-48af-b091-be6b964117ca/volumes/kubernetes.io~projected/kube-api-access-vpbmf:{mountpoint:/var/lib/kubelet/pods/2496f073-7bf0-48af-b091-be6b964117ca/volumes/kubernetes.io~projected/kube-api-access-vpbmf major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2dbd490f-d94c-4f12-9106-a12117deda9d/volumes/kubernetes.io~projected/kube-api-access-d2xzl:{mountpoint:/var/lib/kubelet/pods/2dbd490f-d94c-4f12-9106-a12117deda9d/volumes/kubernetes.io~projected/kube-api-access-d2xzl major:0 minor:280 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6/volumes/kubernetes.io~projected/kube-api-access-4nk9v:{mountpoint:/var/lib/kubelet/pods/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6/volumes/kubernetes.io~projected/kube-api-access-4nk9v major:0 minor:429 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:427 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:428 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3cabcccd-6b1e-4089-a3a6-275558de2bfb/volumes/kubernetes.io~projected/kube-api-access-blx28:{mountpoint:/var/lib/kubelet/pods/3cabcccd-6b1e-4089-a3a6-275558de2bfb/volumes/kubernetes.io~projected/kube-api-access-blx28 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3df036ff-1c00-4ce2-9e86-8acbeedf6c82/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/3df036ff-1c00-4ce2-9e86-8acbeedf6c82/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3df036ff-1c00-4ce2-9e86-8acbeedf6c82/volumes/kubernetes.io~projected/kube-api-access-6r5f6:{mountpoint:/var/lib/kubelet/pods/3df036ff-1c00-4ce2-9e86-8acbeedf6c82/volumes/kubernetes.io~projected/kube-api-access-6r5f6 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3df036ff-1c00-4ce2-9e86-8acbeedf6c82/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/3df036ff-1c00-4ce2-9e86-8acbeedf6c82/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/42aedd5a-da26-4be4-8ed8-01fb25f3ee5f/volumes/kubernetes.io~projected/kube-api-access-kw8lt:{mountpoint:/var/lib/kubelet/pods/42aedd5a-da26-4be4-8ed8-01fb25f3ee5f/volumes/kubernetes.io~projected/kube-api-access-kw8lt major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/42aedd5a-da26-4be4-8ed8-01fb25f3ee5f/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/42aedd5a-da26-4be4-8ed8-01fb25f3ee5f/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4372da07-b518-42df-bbd7-f148396bba6b/volumes/kubernetes.io~projected/kube-api-access-x9vqk:{mountpoint:/var/lib/kubelet/pods/4372da07-b518-42df-bbd7-f148396bba6b/volumes/kubernetes.io~projected/kube-api-access-x9vqk major:0 minor:405 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4372da07-b518-42df-bbd7-f148396bba6b/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/4372da07-b518-42df-bbd7-f148396bba6b/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:402 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4372da07-b518-42df-bbd7-f148396bba6b/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/4372da07-b518-42df-bbd7-f148396bba6b/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:403 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4372da07-b518-42df-bbd7-f148396bba6b/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/4372da07-b518-42df-bbd7-f148396bba6b/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:404 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/454a9788-f96e-453d-a83c-63c550c7d96d/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/454a9788-f96e-453d-a83c-63c550c7d96d/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/59571898-f15b-4b8f-b043-0e5af369c39e/volumes/kubernetes.io~projected/kube-api-access-d7qhc:{mountpoint:/var/lib/kubelet/pods/59571898-f15b-4b8f-b043-0e5af369c39e/volumes/kubernetes.io~projected/kube-api-access-d7qhc major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5c3735e3-95ac-4293-8555-38ee9ead0353/volumes/kubernetes.io~projected/kube-api-access-bq8kj:{mountpoint:/var/lib/kubelet/pods/5c3735e3-95ac-4293-8555-38ee9ead0353/volumes/kubernetes.io~projected/kube-api-access-bq8kj major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6a9841f4-72b0-43fd-8aa7-01393d9a01d1/volumes/kubernetes.io~projected/kube-api-access-g6mc5:{mountpoint:/var/lib/kubelet/pods/6a9841f4-72b0-43fd-8aa7-01393d9a01d1/volumes/kubernetes.io~projected/kube-api-access-g6mc5 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c9c626a-52a7-466e-b744-f4bd79112fd9/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/6c9c626a-52a7-466e-b744-f4bd79112fd9/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74930384-19fd-4c0b-b38f-b2f895f7e6e8/volumes/kubernetes.io~projected/kube-api-access-vpccd:{mountpoint:/var/lib/kubelet/pods/74930384-19fd-4c0b-b38f-b2f895f7e6e8/volumes/kubernetes.io~projected/kube-api-access-vpccd major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74930384-19fd-4c0b-b38f-b2f895f7e6e8/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/74930384-19fd-4c0b-b38f-b2f895f7e6e8/volumes/kubernetes.io~secret/cert major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/83a93696-9f3b-472d-9fc8-92350746df75/volumes/kubernetes.io~projected/kube-api-access-b8gxg:{mountpoint:/var/lib/kubelet/pods/83a93696-9f3b-472d-9fc8-92350746df75/volumes/kubernetes.io~projected/kube-api-access-b8gxg major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/83a93696-9f3b-472d-9fc8-92350746df75/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/83a93696-9f3b-472d-9fc8-92350746df75/volumes/kubernetes.io~secret/metrics-certs major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8ea6188f-9e30-4ab0-b5b3-c99647377e00/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/8ea6188f-9e30-4ab0-b5b3-c99647377e00/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8ea6188f-9e30-4ab0-b5b3-c99647377e00/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/8ea6188f-9e30-4ab0-b5b3-c99647377e00/volumes/kubernetes.io~empty-dir/tmp major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8ea6188f-9e30-4ab0-b5b3-c99647377e00/volumes/kubernetes.io~projected/kube-api-access-b4v76:{mountpoint:/var/lib/kubelet/pods/8ea6188f-9e30-4ab0-b5b3-c99647377e00/volumes/kubernetes.io~projected/kube-api-access-b4v76 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a597eced-76f6-49e8-8176-2e9332188be3/volumes/kubernetes.io~projected/kube-api-access-ttr7d:{mountpoint:/var/lib/kubelet/pods/a597eced-76f6-49e8-8176-2e9332188be3/volumes/kubernetes.io~projected/kube-api-access-ttr7d major:0 minor:363 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a597eced-76f6-49e8-8176-2e9332188be3/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/a597eced-76f6-49e8-8176-2e9332188be3/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:358 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a597eced-76f6-49e8-8176-2e9332188be3/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/a597eced-76f6-49e8-8176-2e9332188be3/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:346 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a597eced-76f6-49e8-8176-2e9332188be3/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/a597eced-76f6-49e8-8176-2e9332188be3/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:359 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a597eced-76f6-49e8-8176-2e9332188be3/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/a597eced-76f6-49e8-8176-2e9332188be3/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:357 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a597eced-76f6-49e8-8176-2e9332188be3/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/a597eced-76f6-49e8-8176-2e9332188be3/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:345 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a597eced-76f6-49e8-8176-2e9332188be3/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/a597eced-76f6-49e8-8176-2e9332188be3/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:347 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a9ca84f8-a6d5-45f4-9137-799e05b12c11/volumes/kubernetes.io~projected/kube-api-access-nbpln:{mountpoint:/var/lib/kubelet/pods/a9ca84f8-a6d5-45f4-9137-799e05b12c11/volumes/kubernetes.io~projected/kube-api-access-nbpln major:0 minor:262 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a9ca84f8-a6d5-45f4-9137-799e05b12c11/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/a9ca84f8-a6d5-45f4-9137-799e05b12c11/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:257 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a9ca84f8-a6d5-45f4-9137-799e05b12c11/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/a9ca84f8-a6d5-45f4-9137-799e05b12c11/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:263 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aed61d9f-eb67-44ae-a094-a12814e42559/volumes/kubernetes.io~projected/kube-api-access-c4dgx:{mountpoint:/var/lib/kubelet/pods/aed61d9f-eb67-44ae-a094-a12814e42559/volumes/kubernetes.io~projected/kube-api-access-c4dgx major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b282542e-5b91-430d-bd5e-03af2e875952/volumes/kubernetes.io~secret/tls-certificates:{mountpoint:/var/lib/kubelet/pods/b282542e-5b91-430d-bd5e-03af2e875952/volumes/kubernetes.io~secret/tls-certificates major:0 minor:246 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2b5bbd1-4e37-4d49-9e28-bdb3c3da309f/volumes/kubernetes.io~projected/kube-api-access-km8jp:{mountpoint:/var/lib/kubelet/pods/c2b5bbd1-4e37-4d49-9e28-bdb3c3da309f/volumes/kubernetes.io~projected/kube-api-access-km8jp major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2b5bbd1-4e37-4d49-9e28-bdb3c3da309f/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/c2b5bbd1-4e37-4d49-9e28-bdb3c3da309f/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ca9f1917-07a8-4ea6-8b59-4f2f99429dfa/volumes/kubernetes.io~projected/kube-api-access-bghfr:{mountpoint:/var/lib/kubelet/pods/ca9f1917-07a8-4ea6-8b59-4f2f99429dfa/volumes/kubernetes.io~projected/kube-api-access-bghfr major:0 minor:301 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ca9f1917-07a8-4ea6-8b59-4f2f99429dfa/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/ca9f1917-07a8-4ea6-8b59-4f2f99429dfa/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ca9f1917-07a8-4ea6-8b59-4f2f99429dfa/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/ca9f1917-07a8-4ea6-8b59-4f2f99429dfa/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:300 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8f2a8e7-53db-4867-bd38-a8d45b15e40d/volumes/kubernetes.io~projected/kube-api-access-dzjnx:{mountpoint:/var/lib/kubelet/pods/d8f2a8e7-53db-4867-bd38-a8d45b15e40d/volumes/kubernetes.io~projected/kube-api-access-dzjnx major:0 minor:422 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8f2a8e7-53db-4867-bd38-a8d45b15e40d/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/d8f2a8e7-53db-4867-bd38-a8d45b15e40d/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:421 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8f2a8e7-53db-4867-bd38-a8d45b15e40d/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/d8f2a8e7-53db-4867-bd38-a8d45b15e40d/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:418 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8f2a8e7-53db-4867-bd38-a8d45b15e40d/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/d8f2a8e7-53db-4867-bd38-a8d45b15e40d/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:419 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8f2a8e7-53db-4867-bd38-a8d45b15e40d/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/d8f2a8e7-53db-4867-bd38-a8d45b15e40d/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:420 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:353 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1/volumes/kubernetes.io~empty-dir/config-out major:0 minor:352 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1/volumes/kubernetes.io~projected/kube-api-access-sw9tn:{mountpoint:/var/lib/kubelet/pods/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1/volumes/kubernetes.io~projected/kube-api-access-sw9tn major:0 minor:356 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1/volumes/kubernetes.io~projected/tls-assets major:0 minor:351 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:344 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1/volumes/kubernetes.io~secret/config-volume major:0 minor:348 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:354 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:355 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:349 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:350 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1/volumes/kubernetes.io~secret/web-config major:0 minor:353 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9f7709e-2664-452f-b70b-f4649675781e/volumes/kubernetes.io~projected/kube-api-access-ptklg:{mountpoint:/var/lib/kubelet/pods/d9f7709e-2664-452f-b70b-f4649675781e/volumes/kubernetes.io~projected/kube-api-access-ptklg major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f0d8d720-b6f1-4600-970f-8191318c3e3e/volumes/kubernetes.io~projected/kube-api-access-lbkld:{mountpoint:/var/lib/kubelet/pods/f0d8d720-b6f1-4600-970f-8191318c3e3e/volumes/kubernetes.io~projected/kube-api-access-lbkld major:0 minor:399 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f0d8d720-b6f1-4600-970f-8191318c3e3e/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/f0d8d720-b6f1-4600-970f-8191318c3e3e/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:395 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/49e8f3e6794d3366141d851434bddce114d3f4b5201a7f713a19bb17947defbb/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/afb334883b36f44891d8ea160ee3d4f2f8d9846969e7bcf002ebc48b72a134d8/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/338a9040a9eaea191e59ee3f0ce07d0cd3812157959c6f286d0dcea9868bf199/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-105:{mountpoint:/var/lib/containers/storage/overlay/5366b26f6d38af65a268a65a821afbba7de428b72ace1b31cb23f983095b7011/merged major:0 minor:105 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/b62e16bb20614db0c065563ddf4a6a63a07b9543da215a80025f6555c7bec71d/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/61a2cbd315957a8e890931504ab20efc52f367081110f50594a240ff321237b3/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/d3f976974ddccb6208add6b72eaf34d58045a79271ca1b608539132db97ca8e8/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/499dfe4c8fa85b6140ff762145bcc2ad89e9048cb0318f76a7376db015c66071/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/2fa2801475dc49c879942cd1bfe749d194e7810c3ec89c4836b4043539bbbe11/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/f17b44eec5273bdf47a47f12bb983709a2a0b84a64e10dffdac5bd051c5af87f/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/61ec61320e057bcd5d6a1a72713b285e762b7bb23a1ac75dfd91d9be3cd8077a/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/3e42bffa4dee7f3d0590ed4a029e9ee76eb1bdffa7289361d119e3a29d24380e/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/fac6237a3872203f61e0be0703b64f60b54ac22117d430a5b91be23b959711f3/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/e5d5fcf708da4a28ef514f6f7e6e642b860b0f4dd5b8aff124fb600bf8e1d8ea/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/0b6ce9e5001a6fc53403aa22b9d90b8dff72060337bbd7e051ee723c338c148b/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/06699e51fdb121dd692477ec08a80af9bdcb7c1f70fa6484b63bf19155a834db/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/d8313e3f45b5a2dd865966b988c8b6e7f30a70b6a92fb6a2a160dadff3263e7d/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/408d3af54298cef89a8095a92c2403149ba0c3785e96942c68a33073fac1d288/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/f9ec718f6c4dfc9f8c25613ba8335e1c1ad7fdb37d1ce3ef10f153db008b7b00/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-184:{mountpoint:/var/lib/containers/storage/overlay/b55ee09099376fe5b8112a535595276833ea8adc52a74886c30c129a1d9f8662/merged major:0 minor:184 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/2a3562df816fbe0760d09dedecfbf327c4805a7062bf3b69d19d37ec18b068f4/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/4f64c392a1b0411be148d03328211ecdc347efcb72546eb92e0b36a538e202f2/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/3608d35a21c02ab874d9541993583bd9d731d2f0bf6e176bccfb33956c739b1d/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/6be25e635a706db9407602e86a80ddfc5027ae3819c39020526c9576f17de31c/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/e2bbf8e605d9153805b2f5512edf463af70bcef02260877980df78879aaca353/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/d13817aec43809a40788f12878e300a7e69594f0f8c5baa96e0fecaa36a817f6/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/42547840d218bd1a952492085289452843fe95633b44011ca41be9515d643e98/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/2b93e955220d2f7ab09dc6e71806bc66abce4cc201c5b30fb8969e5166a48409/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/022567928b3950b6d8b820f68fc2e687353601fbabd2d3a0043525b2e045f006/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/fbf18cc17bc8e51dbf0c77de50425b988cbef1bb21979a52afd6cb572c2e1b14/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/8fa39235c0fdc25277e5a30dee8e366169c66fd516700ea441d10d35b4236d93/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-255:{mountpoint:/var/lib/containers/storage/overlay/2ba23234c64f1532caa1cc95296b29faa9ab12b2c8a1d610c44349184f0d966b/merged major:0 minor:255 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/96d5bf9080ce61f7cdf6520dcf6821a77421cbba51f6ea9d37df04395908934a/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/37ec2b4a062dc5b35cbcd4baa6d7febf089a4b88f285496e677c67b6f686769b/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/913b9c167bb8681a150af977ea5cd78aba67e7d76d56aefbdbda16cc74600411/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/43c88d6147ae71caa22f6a48ac62d10400360a66ba6ccb959369723672dccb1b/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-293:{mountpoint:/var/lib/containers/storage/overlay/c95999c0442872c9815ea551e996bd9e00cbd0a66a1f8d032918319120aaf311/merged major:0 minor:293 fsType:overlay blockSize:0} overlay_0-295:{mountpoint:/var/lib/containers/storage/overlay/22d7f255ba017c1deba8f68fb763c6fa2af3b97a8d6de862a315d3425a307b63/merged major:0 minor:295 fsType:overlay blockSize:0} overlay_0-297:{mountpoint:/var/lib/containers/storage/overlay/eed7b4fcd9a27c6db9fc18113c1cfeb3d6b1af016ae1fe60248075b98df9696f/merged major:0 minor:297 fsType:overlay blockSize:0} overlay_0-314:{mountpoint:/var/lib/containers/storage/overlay/02777610f02fc2be9832bdccd59686aa643ee815bb8bc1d6f0bb056df6e51667/merged major:0 minor:314 fsType:overlay blockSize:0} overlay_0-316:{mountpoint:/var/lib/containers/storage/overlay/ad0f62d60e0ca0b861ec859995fb26004b6be0bc0cdd776c5ea9ecb6f8bbeff5/merged major:0 minor:316 fsType:overlay blockSize:0} overlay_0-318:{mountpoint:/var/lib/containers/storage/overlay/255c010ed4ccf7c5302be5c43e622d615519d497d486d116843fca69f2366380/merged major:0 minor:318 fsType:overlay blockSize:0} overlay_0-320:{mountpoint:/var/lib/containers/storage/overlay/068aac006dd2993fd1e17f4308d2ee435ad73bec6dd145db8facaa2c68d92f1e/merged major:0 minor:320 fsType:overlay blockSize:0} overlay_0-326:{mountpoint:/var/lib/containers/storage/overlay/544bf1014d2e8d63ec296e691999ea3ee5134676f132733811b9a75a5b6dfb14/merged major:0 minor:326 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/e618325d439e0570c6f3d61df334532b7660480e7811d7eb64c1618e14c4a17b/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-333:{mountpoint:/var/lib/containers/storage/overlay/d8f9310befd33f73cc58042def505bb12f4660ee7f9f31e735a6a11da86a5324/merged major:0 minor:333 fsType:overlay blockSize:0} overlay_0-360:{mountpoint:/var/lib/containers/storage/overlay/51a28947a970ddaac74400a0bed114f82fc00c86a000a58dab2b2e74161914b3/merged major:0 minor:360 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/ae6e8f11cd4f4abe0bfe3548c76de9c1d7eead7d7b8de9386c3f21b059f123f1/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-370:{mountpoint:/var/lib/containers/storage/overlay/2f9731798dcb78c15925c277c7c42eaefed82e28faf826a9264638cbfc7c1670/merged major:0 minor:370 fsType:overlay blockSize:0} overlay_0-372:{mountpoint:/var/lib/containers/storage/overlay/e7db16dcfcf71b7cfadffba0acbf8911b358bc2befce2b0cec3b18a9e784c299/merged major:0 minor:372 fsType:overlay blockSize:0} overlay_0-377:{mountpoint:/var/lib/containers/storage/overlay/c5a56030f6ad0f754584fcaf8435bec862c12ee4011bffc61eb9ebb88f4c69d4/merged major:0 minor:377 fsType:overlay blockSize:0} overlay_0-379:{mountpoint:/var/lib/containers/storage/overlay/4a52d76d6f89e8894a28ae50fc9fb2e3a4d061778b157f5d6720e56883ac23dd/merged major:0 minor:379 fsType:overlay blockSize:0} overlay_0-388:{mountpoint:/var/lib/containers/storage/overlay/45962019cc1e16f5aa0e849a6d6daa0d0029e3ff0c8d86537a023a1231e11ec2/merged major:0 minor:388 fsType:overlay blockSize:0} overlay_0-390:{mountpoint:/var/lib/containers/storage/overlay/85fa02209d2fba6b23f2116ba427cd439a6b5b2b4c12c0e9ae9b0c73b3fc19bb/merged major:0 minor:390 fsType:overlay blockSize:0} overlay_0-408:{mountpoint:/var/lib/containers/storage/overlay/88dcd9e1f735578803ac1c94dc2369617c572a9323993d28b6c43589ed06e666/merged major:0 minor:408 fsType:overlay blockSize:0} overlay_0-410:{mountpoint:/var/lib/containers/storage/overlay/841a58a1445c8ae0e1fb522dd3ae0c1e48684beb2916c89277ab5349478af8a2/merged major:0 minor:410 fsType:overlay blockSize:0} overlay_0-412:{mountpoint:/var/lib/containers/storage/overlay/5ba81e7adb69cefa668323fe6d5bab7a4245ed3c5ba2deb114871c3c9faad223/merged major:0 minor:412 fsType:overlay blockSize:0} overlay_0-425:{mountpoint:/var/lib/containers/storage/overlay/2d6b3ebab859b8d6d68f8dfac80548a5705b3dadd8f0c18a45e169d529e98454/merged major:0 minor:425 fsType:overlay blockSize:0} overlay_0-432:{mountpoint:/var/lib/containers/storage/overlay/1c43080a38e505d07201e076fbb108defbf79ead1bf863aaac10c73422720771/merged major:0 minor:432 fsType:overlay blockSize:0} overlay_0-434:{mountpoint:/var/lib/containers/storage/overlay/405d415cde662915033a594600afc570477e08ef5747cdf52ff857694732b863/merged major:0 minor:434 fsType:overlay blockSize:0} overlay_0-436:{mountpoint:/var/lib/containers/storage/overlay/e68e977bfc8d00e65882fe83f9ed2676e1d998a26cd8bf76102b872bc407d285/merged major:0 minor:436 fsType:overlay blockSize:0} overlay_0-442:{mountpoint:/var/lib/containers/storage/overlay/7608a9f38b1acff5e3b1dd6fb13bd08f18c10d4788055907ba78e5e04335fb19/merged major:0 minor:442 fsType:overlay blockSize:0} overlay_0-447:{mountpoint:/var/lib/containers/storage/overlay/670bb8302c333a23ca4858c2e396f4f8dfd7fdc7d97ed68d81149ec83646da69/merged major:0 minor:447 fsType:overlay blockSize:0} overlay_0-452:{mountpoint:/var/lib/containers/storage/overlay/6876576be93252c25e080a22ee4f73149301f283d6e3bc56fff376248a7cb1ca/merged major:0 minor:452 fsType:overlay blockSize:0} overlay_0-458:{mountpoint:/var/lib/containers/storage/overlay/57262b0bce8feb309730403a06c5bfbb85089a3028b829ebaa587605ce260e20/merged major:0 minor:458 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/6c0214c9ae2a2725fcdfe2fc24ddbdcb49c62b325e04e94e87f2083e8c4accc5/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/6c437d8e7affa3dba1461757ee96445a943c8a730d4598a885095932300e3dd8/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/3eb6b58cf3c8d82cce8e81e68e49277217845a58fd4b02beb667b9bfe598a1c2/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/3d633a2c256fe556ff25d7d75fb13c2b817aed4713cfda36870f4f48034aa917/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/3b8647e0c19b811eb963e82c68cd67269df4f9d8250a8922e66d1138f85ced5a/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/cefd64fb1b10bc5343ad588545227dcc20f55b164d4062fbdfde16402ec097d6/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/7d491fce6f18cadf677c193a2cf0e25bec4b89d5a6c8cf42a49293cc0afa5327/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/51eb98a0f64fb2dd4b61b3113acf2767e096c3df53422a8fc72a70354229db19/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/23868ccd8fe0508c655c201bf6be9947918b29e088b797b864b9a872d7c25751/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-97:{mountpoint:/var/lib/containers/storage/overlay/4b1b0c096aa97d0dfdfe6343af3180e9d8e03b165bd870c0590b17bf975b8f9c/merged major:0 minor:97 fsType:overlay blockSize:0}] Oct 30 21:23:36.338388 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.337294 5748 manager.go:217] Machine: {Timestamp:2026-10-30 21:23:36.336780457 +0000 UTC m=+0.079301384 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099690 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b738886855f32b4f9f8b105bc9374 SystemUUID:ec2b7388-8685-5f32-b4f9-f8b105bc9374 BootID:2071c5e8-f6a6-40fc-a852-71159816c8d8 Filesystems:[{Device:/var/lib/kubelet/pods/4372da07-b518-42df-bbd7-f148396bba6b/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:403 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-434 DeviceMajor:0 DeviceMinor:434 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-372 DeviceMajor:0 DeviceMinor:372 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5c3735e3-95ac-4293-8555-38ee9ead0353/volumes/kubernetes.io~projected/kube-api-access-bq8kj DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/74930384-19fd-4c0b-b38f-b2f895f7e6e8/volumes/kubernetes.io~projected/kube-api-access-vpccd DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b594b27c3c927b7ab46df23306276cbd54084dabd3bf99c51864a383dc490c7d/userdata/shm DeviceMajor:0 DeviceMinor:289 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1/volumes/kubernetes.io~projected/kube-api-access-sw9tn DeviceMajor:0 DeviceMinor:356 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4372da07-b518-42df-bbd7-f148396bba6b/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:402 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d8f2a8e7-53db-4867-bd38-a8d45b15e40d/volumes/kubernetes.io~projected/kube-api-access-dzjnx DeviceMajor:0 DeviceMinor:422 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6/volumes/kubernetes.io~projected/kube-api-access-4nk9v DeviceMajor:0 DeviceMinor:429 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/253902cf7fb7a07a5987fe209537eb45f982a10286b6a15f35eacb7027da2cd8/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9dc54f963eddd9f65ef3f9797b5563e449fed9af9fc472616366c4b6d508a75e/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/953276e6631de9919ab79f85e9cad674e90cb10dc00b94a70fea3bf5cef97545/userdata/shm DeviceMajor:0 DeviceMinor:95 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a9ca84f8-a6d5-45f4-9137-799e05b12c11/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:263 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a597eced-76f6-49e8-8176-2e9332188be3/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:358 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-388 DeviceMajor:0 DeviceMinor:388 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/74930384-19fd-4c0b-b38f-b2f895f7e6e8/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/42aedd5a-da26-4be4-8ed8-01fb25f3ee5f/volumes/kubernetes.io~projected/kube-api-access-kw8lt DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ca9f1917-07a8-4ea6-8b59-4f2f99429dfa/volumes/kubernetes.io~projected/kube-api-access-bghfr DeviceMajor:0 DeviceMinor:301 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-410 DeviceMajor:0 DeviceMinor:410 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3cabcccd-6b1e-4089-a3a6-275558de2bfb/volumes/kubernetes.io~projected/kube-api-access-blx28 DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d5ffffe05af64a1646fb52a59c06c8823e1fb52b56b17d704bf19046a036f3b2/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/166e5f35-0987-4eef-917a-a4772acba82f/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:302 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:353 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/26cd448adaaa27c53605b8c3447aadf82556ccb34651e6e39acc6b4688aaf96a/userdata/shm DeviceMajor:0 DeviceMinor:400 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8ea6188f-9e30-4ab0-b5b3-c99647377e00/volumes/kubernetes.io~projected/kube-api-access-b4v76 DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e1b78fb648faa00672c441f6ea61fae944f1bcd635af480835decf0a41cc8f60/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-105 DeviceMajor:0 DeviceMinor:105 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d8f2a8e7-53db-4867-bd38-a8d45b15e40d/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:418 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-425 DeviceMajor:0 DeviceMinor:425 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-436 DeviceMajor:0 DeviceMinor:436 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-447 DeviceMajor:0 DeviceMinor:447 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/21dac7bbe759d29d25155018511b5532b1b78d6e336d26183d4d7505e28570ab/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c2b5bbd1-4e37-4d49-9e28-bdb3c3da309f/volumes/kubernetes.io~projected/kube-api-access-km8jp DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a283ea7030cc46493344d2445e51bba7633bc33dd6cd07a5f47a6bdef80a718b/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/16851feff42b6b44a4b4f281e1734b22181590f88d68cd1bb7f73d0622bd9703/userdata/shm DeviceMajor:0 DeviceMinor:251 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:344 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2a4f46994282538de689d63e23d77357bf6940b5effeecfdd9ee73ecf275991b/userdata/shm DeviceMajor:0 DeviceMinor:362 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2a6c6abcf5dc8b26d1d6d6f868f70a22fb169f9647327bab20ca6c3646432656/userdata/shm DeviceMajor:0 DeviceMinor:276 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:351 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-360 DeviceMajor:0 DeviceMinor:360 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-458 DeviceMajor:0 DeviceMinor:458 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2e8ccef945c9b2ab26da19f76a836a73756872b7554e0a67985accacdcb37fe9/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b092a4deb6fd47f6577074ba533fcd9831761318517be661b91db0c2191c919f/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-97 DeviceMajor:0 DeviceMinor:97 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2496f073-7bf0-48af-b091-be6b964117ca/volumes/kubernetes.io~projected/kube-api-access-vpbmf DeviceMajor:0 DeviceMinor:286 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/166e5f35-0987-4eef-917a-a4772acba82f/volumes/kubernetes.io~projected/kube-api-access-459jm DeviceMajor:0 DeviceMinor:304 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-314 DeviceMajor:0 DeviceMinor:314 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/454a9788-f96e-453d-a83c-63c550c7d96d/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6a9841f4-72b0-43fd-8aa7-01393d9a01d1/volumes/kubernetes.io~projected/kube-api-access-g6mc5 DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8ea6188f-9e30-4ab0-b5b3-c99647377e00/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-295 DeviceMajor:0 DeviceMinor:295 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/839bfab96e7f8e0962d2a288948ea4e459936b713d5626d73efb3434d0f2d12a/userdata/shm DeviceMajor:0 DeviceMinor:312 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6b2a31073c7737830fb55f6fe12435407df6541f64cf11219ac235d3ab7db600/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7b1cb4f6a870992a27f8a7c72f56ffe89d0e2b5519d6fcad4ff407278af0bc83/userdata/shm DeviceMajor:0 DeviceMinor:291 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-316 DeviceMajor:0 DeviceMinor:316 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:353 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-379 DeviceMajor:0 DeviceMinor:379 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-412 DeviceMajor:0 DeviceMinor:412 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:427 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/42aedd5a-da26-4be4-8ed8-01fb25f3ee5f/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-297 DeviceMajor:0 DeviceMinor:297 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a597eced-76f6-49e8-8176-2e9332188be3/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:359 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f0d8d720-b6f1-4600-970f-8191318c3e3e/volumes/kubernetes.io~projected/kube-api-access-lbkld DeviceMajor:0 DeviceMinor:399 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-442 DeviceMajor:0 DeviceMinor:442 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aed61d9f-eb67-44ae-a094-a12814e42559/volumes/kubernetes.io~projected/kube-api-access-c4dgx DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:307 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-326 DeviceMajor:0 DeviceMinor:326 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ac3b2f70532aadcd84f622494b52343b96ea6fa8e52d777db57d766979f199fb/userdata/shm DeviceMajor:0 DeviceMinor:339 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d8f2a8e7-53db-4867-bd38-a8d45b15e40d/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:421 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/90b5c7445469629806ce6590748554079f89b212ed99dfc9f33c2f39aea5951c/userdata/shm DeviceMajor:0 DeviceMinor:430 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ed963bfa9bbb231b5afbfba50c8b3914a24934b413908bda4c6f037527000a01/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/313b3e77bdc93435cc744be37f449318264b5b426dbe34048426a475aeadafcb/userdata/shm DeviceMajor:0 DeviceMinor:287 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:348 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a597eced-76f6-49e8-8176-2e9332188be3/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:346 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a597eced-76f6-49e8-8176-2e9332188be3/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:347 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f0d8d720-b6f1-4600-970f-8191318c3e3e/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:395 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4372da07-b518-42df-bbd7-f148396bba6b/volumes/kubernetes.io~projected/kube-api-access-x9vqk DeviceMajor:0 DeviceMinor:405 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8ea6188f-9e30-4ab0-b5b3-c99647377e00/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-184 DeviceMajor:0 DeviceMinor:184 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-293 DeviceMajor:0 DeviceMinor:293 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ca9f1917-07a8-4ea6-8b59-4f2f99429dfa/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:299 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d8f2a8e7-53db-4867-bd38-a8d45b15e40d/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:420 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/166e5f35-0987-4eef-917a-a4772acba82f/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:303 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:308 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:349 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a597eced-76f6-49e8-8176-2e9332188be3/volumes/kubernetes.io~projected/kube-api-access-ttr7d DeviceMajor:0 DeviceMinor:363 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f6a7fe29d8810324e1e8f160b24c936f3766ae93c0e454b9d78eef9f84b05a86/userdata/shm DeviceMajor:0 DeviceMinor:406 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a9ca84f8-a6d5-45f4-9137-799e05b12c11/volumes/kubernetes.io~projected/kube-api-access-nbpln DeviceMajor:0 DeviceMinor:262 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-333 DeviceMajor:0 DeviceMinor:333 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:350 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b4c99a19e2a768b0995b24beda1a0329650dd8b1feed61600397baf6c1a0abe2/userdata/shm DeviceMajor:0 DeviceMinor:264 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4372da07-b518-42df-bbd7-f148396bba6b/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:404 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:352 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a597eced-76f6-49e8-8176-2e9332188be3/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:357 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-390 DeviceMajor:0 DeviceMinor:390 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-432 DeviceMajor:0 DeviceMinor:432 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/be8bf0429e5f966f001d3467b942c1f2c894b8e25c67bdba0df0bce3b28e9b59/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d9f7709e-2664-452f-b70b-f4649675781e/volumes/kubernetes.io~projected/kube-api-access-ptklg DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3df036ff-1c00-4ce2-9e86-8acbeedf6c82/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-255 DeviceMajor:0 DeviceMinor:255 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:354 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ca9f1917-07a8-4ea6-8b59-4f2f99429dfa/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:300 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:428 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b282542e-5b91-430d-bd5e-03af2e875952/volumes/kubernetes.io~secret/tls-certificates DeviceMajor:0 DeviceMinor:246 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/83a93696-9f3b-472d-9fc8-92350746df75/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:284 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979/volumes/kubernetes.io~projected/kube-api-access-hs2w6 DeviceMajor:0 DeviceMinor:309 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-408 DeviceMajor:0 DeviceMinor:408 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ff15d90480ca95d335939e1f2e01526ffd17ff5dc66f00301b363e9c71972421/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6d53dc7170d4956d3c83148a21dfb0a03609822ffe6b368653e1331804f3df7d/userdata/shm DeviceMajor:0 DeviceMinor:305 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ee9f13f99dd386c9fdec54d4ce3a925258a068ea279cbfbc01ee5a7d8534a773/userdata/shm DeviceMajor:0 DeviceMinor:423 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/83a93696-9f3b-472d-9fc8-92350746df75/volumes/kubernetes.io~projected/kube-api-access-b8gxg DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-318 DeviceMajor:0 DeviceMinor:318 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-452 DeviceMajor:0 DeviceMinor:452 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3df036ff-1c00-4ce2-9e86-8acbeedf6c82/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/59571898-f15b-4b8f-b043-0e5af369c39e/volumes/kubernetes.io~projected/kube-api-access-d7qhc DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2dbd490f-d94c-4f12-9106-a12117deda9d/volumes/kubernetes.io~projected/kube-api-access-d2xzl DeviceMajor:0 DeviceMinor:280 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/381f7b82d8dc76d1ea1c32732ba49780f515bf1815ff83c89f5870338b25d09c/userdata/shm DeviceMajor:0 DeviceMinor:310 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-370 DeviceMajor:0 DeviceMinor:370 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-377 DeviceMajor:0 DeviceMinor:377 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d8f2a8e7-53db-4867-bd38-a8d45b15e40d/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:419 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/3df036ff-1c00-4ce2-9e86-8acbeedf6c82/volumes/kubernetes.io~projected/kube-api-access-6r5f6 DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-320 DeviceMajor:0 DeviceMinor:320 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:355 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a597eced-76f6-49e8-8176-2e9332188be3/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:345 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a9ca84f8-a6d5-45f4-9137-799e05b12c11/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:257 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6c9c626a-52a7-466e-b744-f4bd79112fd9/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:285 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c2b5bbd1-4e37-4d49-9e28-bdb3c3da309f/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:16851feff42b6b4 MacAddress:1e:68:d7:87:14:1a Speed:10000 Mtu:8901} {Name:26cd448adaaa27c MacAddress:06:98:1e:88:33:eb Speed:10000 Mtu:8901} {Name:2a4f46994282538 MacAddress:b2:3f:a4:c6:17:cb Speed:10000 Mtu:8901} {Name:2a6c6abcf5dc8b2 MacAddress:ea:6b:f7:b1:30:b3 Speed:10000 Mtu:8901} {Name:313b3e77bdc9343 MacAddress:8a:ea:cf:b0:93:63 Speed:10000 Mtu:8901} {Name:381f7b82d8dc76d MacAddress:46:07:43:e9:f0:0a Speed:10000 Mtu:8901} {Name:6d53dc7170d4956 MacAddress:da:40:47:a0:29:68 Speed:10000 Mtu:8901} {Name:7b1cb4f6a870992 MacAddress:96:13:7a:27:c3:3e Speed:10000 Mtu:8901} {Name:90b5c7445469629 MacAddress:12:1e:8a:aa:ac:b7 Speed:10000 Mtu:8901} {Name:a283ea7030cc464 MacAddress:96:88:38:26:3d:6f Speed:10000 Mtu:8901} {Name:ac3b2f70532aadc MacAddress:0e:aa:82:cf:fd:62 Speed:10000 Mtu:8901} {Name:b4c99a19e2a768b MacAddress:06:44:7c:d8:1e:62 Speed:10000 Mtu:8901} {Name:b594b27c3c927b7 MacAddress:6e:0f:3d:ea:2e:c4 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:68:6d:87:0b:2d Speed:0 Mtu:9001} {Name:br-int MacAddress:1e:54:5b:7d:93:41 Speed:0 Mtu:8901} {Name:d5ffffe05af64a1 MacAddress:6a:33:cb:7d:6a:e3 Speed:10000 Mtu:8901} {Name:ee9f13f99dd386c MacAddress:36:41:83:49:c2:d9 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:68:6d:87:0b:2d Speed:0 Mtu:9001} {Name:f6a7fe29d881032 MacAddress:26:68:d9:5e:54:1a Speed:10000 Mtu:8901} {Name:ff15d90480ca95d MacAddress:de:6a:4f:b9:59:8b Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:ca:18:ca:30:6a:04 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:f2:aa:c7:d3:cd:8f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 21:23:36.338388 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.337850 5748 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 21:23:36.338388 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.337932 5748 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 21:23:36.338388 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.338246 5748 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 21:23:36.338832 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.338277 5748 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-61.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 21:23:36.338832 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.338472 5748 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 21:23:36.338832 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.338485 5748 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 21:23:36.338832 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.338499 5748 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:23:36.338832 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.338525 5748 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:23:36.338832 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.338647 5748 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 21:23:36.339124 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.338849 5748 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 21:23:36.339124 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.339033 5748 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 21:23:36.339124 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.339077 5748 kubelet.go:493] "Attempting to sync node with API server" Oct 30 21:23:36.339124 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.339090 5748 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 21:23:36.339124 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.339106 5748 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 21:23:36.339124 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.339119 5748 kubelet.go:405] "Adding apiserver pod source" Oct 30 21:23:36.339357 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.339129 5748 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 21:23:36.341040 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.341005 5748 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 21:23:36.341040 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.341032 5748 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:23:36.343141 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.343116 5748 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 21:23:36.362105 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.362078 5748 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 21:23:36.362105 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.362098 5748 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:23:36.362312 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.362266 5748 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 21:23:36.362545 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.362530 5748 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 21:23:36.362593 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.362563 5748 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 21:23:36.362684 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.362675 5748 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 21:23:36.362714 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.362690 5748 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 21:23:36.362714 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.362697 5748 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 21:23:36.362714 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.362704 5748 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 21:23:36.362714 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.362710 5748 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 21:23:36.362810 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.362717 5748 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 21:23:36.362810 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.362723 5748 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 21:23:36.362810 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.362729 5748 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 21:23:36.362810 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.362737 5748 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 21:23:36.362810 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.362743 5748 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 21:23:36.362810 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.362759 5748 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 21:23:36.362810 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.362769 5748 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 21:23:36.362810 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.362793 5748 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 21:23:36.362810 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.362800 5748 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 21:23:36.366546 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.366533 5748 server.go:1263] "Started kubelet" Oct 30 21:23:36.366704 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.366639 5748 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 21:23:36.366743 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.366735 5748 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 21:23:36.366839 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.366813 5748 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 21:23:36.367106 ip-10-0-143-61 systemd[1]: Started Kubernetes Kubelet. Oct 30 21:23:36.367941 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.367353 5748 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 21:23:36.368268 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.368252 5748 server.go:317] "Adding debug handlers to kubelet server" Oct 30 21:23:36.370985 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.370964 5748 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 21:23:36.371247 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.371231 5748 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 21:23:36.371247 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.371247 5748 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 21:23:36.371434 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.371349 5748 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 21:23:36.371434 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:23:36.371408 5748 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-61.us-west-1.compute.internal\" not found" Oct 30 21:23:36.372744 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.371887 5748 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 21:23:36.372744 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.372155 5748 factory.go:55] Registering systemd factory Oct 30 21:23:36.372744 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.372176 5748 factory.go:223] Registration of the systemd container factory successfully Oct 30 21:23:36.372943 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.372736 5748 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:17:22 +0000 UTC" deadline="2028-05-13 22:03:27.626871433 +0000 UTC" Oct 30 21:23:36.372943 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.372763 5748 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13464h39m51.254111585s" Oct 30 21:23:36.376073 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.376048 5748 factory.go:153] Registering CRI-O factory Oct 30 21:23:36.376073 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.376078 5748 factory.go:223] Registration of the crio container factory successfully Oct 30 21:23:36.376211 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.376161 5748 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 21:23:36.376211 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.376188 5748 factory.go:103] Registering Raw factory Oct 30 21:23:36.376211 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.376208 5748 manager.go:1199] Started watching for new ooms in manager Oct 30 21:23:36.376987 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.376934 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979" volumeName="kubernetes.io/projected/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979-kube-api-access-hs2w6" seLinuxMountContext="" Oct 30 21:23:36.377110 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.376994 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3df036ff-1c00-4ce2-9e86-8acbeedf6c82" volumeName="kubernetes.io/projected/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-kube-api-access-6r5f6" seLinuxMountContext="" Oct 30 21:23:36.377110 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.377016 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42aedd5a-da26-4be4-8ed8-01fb25f3ee5f" volumeName="kubernetes.io/projected/42aedd5a-da26-4be4-8ed8-01fb25f3ee5f-kube-api-access-kw8lt" seLinuxMountContext="" Oct 30 21:23:36.377110 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.377043 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4372da07-b518-42df-bbd7-f148396bba6b" volumeName="kubernetes.io/secret/4372da07-b518-42df-bbd7-f148396bba6b-secret-metrics-server-client-certs" seLinuxMountContext="" Oct 30 21:23:36.377110 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.377060 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2b5bbd1-4e37-4d49-9e28-bdb3c3da309f" volumeName="kubernetes.io/projected/c2b5bbd1-4e37-4d49-9e28-bdb3c3da309f-kube-api-access-km8jp" seLinuxMountContext="" Oct 30 21:23:36.377295 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.377080 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83a93696-9f3b-472d-9fc8-92350746df75" volumeName="kubernetes.io/secret/83a93696-9f3b-472d-9fc8-92350746df75-metrics-certs" seLinuxMountContext="" Oct 30 21:23:36.377295 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.377163 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2dbd490f-d94c-4f12-9106-a12117deda9d" volumeName="kubernetes.io/empty-dir/2dbd490f-d94c-4f12-9106-a12117deda9d-tmp" seLinuxMountContext="" Oct 30 21:23:36.377295 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.377177 5748 manager.go:319] Starting recovery of all containers Oct 30 21:23:36.377295 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.377185 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30b586ee-8d8a-4e95-b9a3-64f3b09c77e6" volumeName="kubernetes.io/configmap/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-service-ca" seLinuxMountContext="" Oct 30 21:23:36.377295 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.377217 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30b586ee-8d8a-4e95-b9a3-64f3b09c77e6" volumeName="kubernetes.io/projected/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-kube-api-access-4nk9v" seLinuxMountContext="" Oct 30 21:23:36.377295 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.377235 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3cabcccd-6b1e-4089-a3a6-275558de2bfb" volumeName="kubernetes.io/configmap/3cabcccd-6b1e-4089-a3a6-275558de2bfb-multus-daemon-config" seLinuxMountContext="" Oct 30 21:23:36.377295 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.377256 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3df036ff-1c00-4ce2-9e86-8acbeedf6c82" volumeName="kubernetes.io/configmap/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-env-overrides" seLinuxMountContext="" Oct 30 21:23:36.377295 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.377275 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42aedd5a-da26-4be4-8ed8-01fb25f3ee5f" volumeName="kubernetes.io/configmap/42aedd5a-da26-4be4-8ed8-01fb25f3ee5f-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 21:23:36.377643 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.377304 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4372da07-b518-42df-bbd7-f148396bba6b" volumeName="kubernetes.io/configmap/4372da07-b518-42df-bbd7-f148396bba6b-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 21:23:36.377643 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.377320 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ea6188f-9e30-4ab0-b5b3-c99647377e00" volumeName="kubernetes.io/empty-dir/8ea6188f-9e30-4ab0-b5b3-c99647377e00-etc-tuned" seLinuxMountContext="" Oct 30 21:23:36.378436 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.377345 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9ca84f8-a6d5-45f4-9137-799e05b12c11" volumeName="kubernetes.io/secret/a9ca84f8-a6d5-45f4-9137-799e05b12c11-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 21:23:36.378517 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.378464 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1" volumeName="kubernetes.io/configmap/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-metrics-client-ca" seLinuxMountContext="" Oct 30 21:23:36.378517 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.378484 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1" volumeName="kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-cluster-tls-config" seLinuxMountContext="" Oct 30 21:23:36.378517 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.378499 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979" volumeName="kubernetes.io/secret/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979-node-exporter-tls" seLinuxMountContext="" Oct 30 21:23:36.378647 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.378523 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3df036ff-1c00-4ce2-9e86-8acbeedf6c82" volumeName="kubernetes.io/configmap/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-ovnkube-script-lib" seLinuxMountContext="" Oct 30 21:23:36.378647 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.378539 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42aedd5a-da26-4be4-8ed8-01fb25f3ee5f" volumeName="kubernetes.io/empty-dir/42aedd5a-da26-4be4-8ed8-01fb25f3ee5f-data-volume" seLinuxMountContext="" Oct 30 21:23:36.378647 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.378559 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a597eced-76f6-49e8-8176-2e9332188be3" volumeName="kubernetes.io/projected/a597eced-76f6-49e8-8176-2e9332188be3-kube-api-access-ttr7d" seLinuxMountContext="" Oct 30 21:23:36.378647 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.378584 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a597eced-76f6-49e8-8176-2e9332188be3" volumeName="kubernetes.io/secret/a597eced-76f6-49e8-8176-2e9332188be3-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Oct 30 21:23:36.378647 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.378605 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2b5bbd1-4e37-4d49-9e28-bdb3c3da309f" volumeName="kubernetes.io/configmap/c2b5bbd1-4e37-4d49-9e28-bdb3c3da309f-config-volume" seLinuxMountContext="" Oct 30 21:23:36.378647 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.378625 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8f2a8e7-53db-4867-bd38-a8d45b15e40d" volumeName="kubernetes.io/configmap/d8f2a8e7-53db-4867-bd38-a8d45b15e40d-telemeter-trusted-ca-bundle" seLinuxMountContext="" Oct 30 21:23:36.378889 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.378648 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8f2a8e7-53db-4867-bd38-a8d45b15e40d" volumeName="kubernetes.io/projected/d8f2a8e7-53db-4867-bd38-a8d45b15e40d-kube-api-access-dzjnx" seLinuxMountContext="" Oct 30 21:23:36.378889 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.378671 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="166e5f35-0987-4eef-917a-a4772acba82f" volumeName="kubernetes.io/configmap/166e5f35-0987-4eef-917a-a4772acba82f-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Oct 30 21:23:36.378889 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.378688 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74930384-19fd-4c0b-b38f-b2f895f7e6e8" volumeName="kubernetes.io/secret/74930384-19fd-4c0b-b38f-b2f895f7e6e8-cert" seLinuxMountContext="" Oct 30 21:23:36.378889 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.378702 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9ca84f8-a6d5-45f4-9137-799e05b12c11" volumeName="kubernetes.io/configmap/a9ca84f8-a6d5-45f4-9137-799e05b12c11-metrics-client-ca" seLinuxMountContext="" Oct 30 21:23:36.378889 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.378733 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8f2a8e7-53db-4867-bd38-a8d45b15e40d" volumeName="kubernetes.io/secret/d8f2a8e7-53db-4867-bd38-a8d45b15e40d-federate-client-tls" seLinuxMountContext="" Oct 30 21:23:36.378889 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.378751 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1" volumeName="kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-web-config" seLinuxMountContext="" Oct 30 21:23:36.378889 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.378762 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="454a9788-f96e-453d-a83c-63c550c7d96d" volumeName="kubernetes.io/secret/454a9788-f96e-453d-a83c-63c550c7d96d-agent-certs" seLinuxMountContext="" Oct 30 21:23:36.378889 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.378776 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a9841f4-72b0-43fd-8aa7-01393d9a01d1" volumeName="kubernetes.io/configmap/6a9841f4-72b0-43fd-8aa7-01393d9a01d1-cni-binary-copy" seLinuxMountContext="" Oct 30 21:23:36.378889 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.378788 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ea6188f-9e30-4ab0-b5b3-c99647377e00" volumeName="kubernetes.io/empty-dir/8ea6188f-9e30-4ab0-b5b3-c99647377e00-tmp" seLinuxMountContext="" Oct 30 21:23:36.378889 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.378800 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a597eced-76f6-49e8-8176-2e9332188be3" volumeName="kubernetes.io/empty-dir/a597eced-76f6-49e8-8176-2e9332188be3-tmp" seLinuxMountContext="" Oct 30 21:23:36.378889 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.378812 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a597eced-76f6-49e8-8176-2e9332188be3" volumeName="kubernetes.io/secret/a597eced-76f6-49e8-8176-2e9332188be3-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Oct 30 21:23:36.378889 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.378825 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1" volumeName="kubernetes.io/empty-dir/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-config-out" seLinuxMountContext="" Oct 30 21:23:36.378889 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.378837 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1" volumeName="kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 21:23:36.378889 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.378849 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30b586ee-8d8a-4e95-b9a3-64f3b09c77e6" volumeName="kubernetes.io/empty-dir/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-tmp" seLinuxMountContext="" Oct 30 21:23:36.378889 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.378860 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59571898-f15b-4b8f-b043-0e5af369c39e" volumeName="kubernetes.io/projected/59571898-f15b-4b8f-b043-0e5af369c39e-kube-api-access-d7qhc" seLinuxMountContext="" Oct 30 21:23:36.378889 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.378873 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c3735e3-95ac-4293-8555-38ee9ead0353" volumeName="kubernetes.io/configmap/5c3735e3-95ac-4293-8555-38ee9ead0353-iptables-alerter-script" seLinuxMountContext="" Oct 30 21:23:36.378889 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.378913 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a9841f4-72b0-43fd-8aa7-01393d9a01d1" volumeName="kubernetes.io/projected/6a9841f4-72b0-43fd-8aa7-01393d9a01d1-kube-api-access-g6mc5" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.378927 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a597eced-76f6-49e8-8176-2e9332188be3" volumeName="kubernetes.io/secret/a597eced-76f6-49e8-8176-2e9332188be3-secret-thanos-querier-tls" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.378944 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9ca84f8-a6d5-45f4-9137-799e05b12c11" volumeName="kubernetes.io/projected/a9ca84f8-a6d5-45f4-9137-799e05b12c11-kube-api-access-nbpln" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.378955 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aed61d9f-eb67-44ae-a094-a12814e42559" volumeName="kubernetes.io/projected/aed61d9f-eb67-44ae-a094-a12814e42559-kube-api-access-c4dgx" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.378966 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b282542e-5b91-430d-bd5e-03af2e875952" volumeName="kubernetes.io/secret/b282542e-5b91-430d-bd5e-03af2e875952-tls-certificates" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.378981 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="166e5f35-0987-4eef-917a-a4772acba82f" volumeName="kubernetes.io/configmap/166e5f35-0987-4eef-917a-a4772acba82f-metrics-client-ca" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.378993 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2dbd490f-d94c-4f12-9106-a12117deda9d" volumeName="kubernetes.io/projected/2dbd490f-d94c-4f12-9106-a12117deda9d-kube-api-access-d2xzl" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.379007 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30b586ee-8d8a-4e95-b9a3-64f3b09c77e6" volumeName="kubernetes.io/secret/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-console-oauth-config" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.379018 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3cabcccd-6b1e-4089-a3a6-275558de2bfb" volumeName="kubernetes.io/projected/3cabcccd-6b1e-4089-a3a6-275558de2bfb-kube-api-access-blx28" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.379031 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4372da07-b518-42df-bbd7-f148396bba6b" volumeName="kubernetes.io/empty-dir/4372da07-b518-42df-bbd7-f148396bba6b-audit-log" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.379047 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca9f1917-07a8-4ea6-8b59-4f2f99429dfa" volumeName="kubernetes.io/configmap/ca9f1917-07a8-4ea6-8b59-4f2f99429dfa-metrics-client-ca" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.379059 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca9f1917-07a8-4ea6-8b59-4f2f99429dfa" volumeName="kubernetes.io/secret/ca9f1917-07a8-4ea6-8b59-4f2f99429dfa-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.379071 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8f2a8e7-53db-4867-bd38-a8d45b15e40d" volumeName="kubernetes.io/configmap/d8f2a8e7-53db-4867-bd38-a8d45b15e40d-serving-certs-ca-bundle" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.379084 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30b586ee-8d8a-4e95-b9a3-64f3b09c77e6" volumeName="kubernetes.io/secret/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-console-serving-cert" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.379096 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59571898-f15b-4b8f-b043-0e5af369c39e" volumeName="kubernetes.io/configmap/59571898-f15b-4b8f-b043-0e5af369c39e-serviceca" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.379107 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9ca84f8-a6d5-45f4-9137-799e05b12c11" volumeName="kubernetes.io/secret/a9ca84f8-a6d5-45f4-9137-799e05b12c11-prometheus-operator-tls" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.379128 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2b5bbd1-4e37-4d49-9e28-bdb3c3da309f" volumeName="kubernetes.io/secret/c2b5bbd1-4e37-4d49-9e28-bdb3c3da309f-metrics-tls" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.379142 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8f2a8e7-53db-4867-bd38-a8d45b15e40d" volumeName="kubernetes.io/configmap/d8f2a8e7-53db-4867-bd38-a8d45b15e40d-metrics-client-ca" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.379155 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8f2a8e7-53db-4867-bd38-a8d45b15e40d" volumeName="kubernetes.io/secret/d8f2a8e7-53db-4867-bd38-a8d45b15e40d-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.379167 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1" volumeName="kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-secret-alertmanager-main-tls" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.379179 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1" volumeName="kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.379191 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979" volumeName="kubernetes.io/configmap/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979-metrics-client-ca" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.379204 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979" volumeName="kubernetes.io/configmap/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.379217 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30b586ee-8d8a-4e95-b9a3-64f3b09c77e6" volumeName="kubernetes.io/configmap/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-oauth-serving-cert" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.379230 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca9f1917-07a8-4ea6-8b59-4f2f99429dfa" volumeName="kubernetes.io/projected/ca9f1917-07a8-4ea6-8b59-4f2f99429dfa-kube-api-access-bghfr" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.379243 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8f2a8e7-53db-4867-bd38-a8d45b15e40d" volumeName="kubernetes.io/secret/d8f2a8e7-53db-4867-bd38-a8d45b15e40d-telemeter-client-tls" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.379256 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0d8d720-b6f1-4600-970f-8191318c3e3e" volumeName="kubernetes.io/secret/f0d8d720-b6f1-4600-970f-8191318c3e3e-monitoring-plugin-cert" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.379268 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="166e5f35-0987-4eef-917a-a4772acba82f" volumeName="kubernetes.io/empty-dir/166e5f35-0987-4eef-917a-a4772acba82f-volume-directive-shadow" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.379281 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979" volumeName="kubernetes.io/secret/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.379294 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42aedd5a-da26-4be4-8ed8-01fb25f3ee5f" volumeName="kubernetes.io/secret/42aedd5a-da26-4be4-8ed8-01fb25f3ee5f-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.379306 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c3735e3-95ac-4293-8555-38ee9ead0353" volumeName="kubernetes.io/projected/5c3735e3-95ac-4293-8555-38ee9ead0353-kube-api-access-bq8kj" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.379319 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a9841f4-72b0-43fd-8aa7-01393d9a01d1" volumeName="kubernetes.io/configmap/6a9841f4-72b0-43fd-8aa7-01393d9a01d1-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.379333 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a597eced-76f6-49e8-8176-2e9332188be3" volumeName="kubernetes.io/configmap/a597eced-76f6-49e8-8176-2e9332188be3-metrics-client-ca" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.379345 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a597eced-76f6-49e8-8176-2e9332188be3" volumeName="kubernetes.io/secret/a597eced-76f6-49e8-8176-2e9332188be3-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.379360 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2b5bbd1-4e37-4d49-9e28-bdb3c3da309f" volumeName="kubernetes.io/empty-dir/c2b5bbd1-4e37-4d49-9e28-bdb3c3da309f-tmp-dir" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.379371 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3df036ff-1c00-4ce2-9e86-8acbeedf6c82" volumeName="kubernetes.io/configmap/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-ovnkube-config" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.379383 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4372da07-b518-42df-bbd7-f148396bba6b" volumeName="kubernetes.io/configmap/4372da07-b518-42df-bbd7-f148396bba6b-metrics-server-audit-profiles" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.379395 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a597eced-76f6-49e8-8176-2e9332188be3" volumeName="kubernetes.io/secret/a597eced-76f6-49e8-8176-2e9332188be3-secret-grpc-tls" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.379407 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a597eced-76f6-49e8-8176-2e9332188be3" volumeName="kubernetes.io/secret/a597eced-76f6-49e8-8176-2e9332188be3-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.379418 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca9f1917-07a8-4ea6-8b59-4f2f99429dfa" volumeName="kubernetes.io/secret/ca9f1917-07a8-4ea6-8b59-4f2f99429dfa-openshift-state-metrics-tls" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.379430 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8f2a8e7-53db-4867-bd38-a8d45b15e40d" volumeName="kubernetes.io/secret/d8f2a8e7-53db-4867-bd38-a8d45b15e40d-secret-telemeter-client" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.379442 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1" volumeName="kubernetes.io/configmap/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.379454 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1" volumeName="kubernetes.io/projected/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-kube-api-access-sw9tn" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.379466 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="166e5f35-0987-4eef-917a-a4772acba82f" volumeName="kubernetes.io/secret/166e5f35-0987-4eef-917a-a4772acba82f-kube-state-metrics-tls" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.379478 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30b586ee-8d8a-4e95-b9a3-64f3b09c77e6" volumeName="kubernetes.io/configmap/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-console-config" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.379489 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4372da07-b518-42df-bbd7-f148396bba6b" volumeName="kubernetes.io/secret/4372da07-b518-42df-bbd7-f148396bba6b-secret-metrics-server-tls" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.379500 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="454a9788-f96e-453d-a83c-63c550c7d96d" volumeName="kubernetes.io/configmap/454a9788-f96e-453d-a83c-63c550c7d96d-konnectivity-ca" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.379511 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1" volumeName="kubernetes.io/empty-dir/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-alertmanager-main-db" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.379523 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="166e5f35-0987-4eef-917a-a4772acba82f" volumeName="kubernetes.io/projected/166e5f35-0987-4eef-917a-a4772acba82f-kube-api-access-459jm" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.379537 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2496f073-7bf0-48af-b091-be6b964117ca" volumeName="kubernetes.io/projected/2496f073-7bf0-48af-b091-be6b964117ca-kube-api-access-vpbmf" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.379551 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3cabcccd-6b1e-4089-a3a6-275558de2bfb" volumeName="kubernetes.io/configmap/3cabcccd-6b1e-4089-a3a6-275558de2bfb-cni-binary-copy" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.379563 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3df036ff-1c00-4ce2-9e86-8acbeedf6c82" volumeName="kubernetes.io/secret/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.379574 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1" volumeName="kubernetes.io/projected/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-tls-assets" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.379588 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9f7709e-2664-452f-b70b-f4649675781e" volumeName="kubernetes.io/projected/d9f7709e-2664-452f-b70b-f4649675781e-kube-api-access-ptklg" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.379600 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0d8d720-b6f1-4600-970f-8191318c3e3e" volumeName="kubernetes.io/projected/f0d8d720-b6f1-4600-970f-8191318c3e3e-kube-api-access-lbkld" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.379613 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ea6188f-9e30-4ab0-b5b3-c99647377e00" volumeName="kubernetes.io/projected/8ea6188f-9e30-4ab0-b5b3-c99647377e00-kube-api-access-b4v76" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.379626 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83a93696-9f3b-472d-9fc8-92350746df75" volumeName="kubernetes.io/projected/83a93696-9f3b-472d-9fc8-92350746df75-kube-api-access-b8gxg" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.379638 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4372da07-b518-42df-bbd7-f148396bba6b" volumeName="kubernetes.io/projected/4372da07-b518-42df-bbd7-f148396bba6b-kube-api-access-x9vqk" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.379650 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4372da07-b518-42df-bbd7-f148396bba6b" volumeName="kubernetes.io/secret/4372da07-b518-42df-bbd7-f148396bba6b-client-ca-bundle" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.379662 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a9841f4-72b0-43fd-8aa7-01393d9a01d1" volumeName="kubernetes.io/configmap/6a9841f4-72b0-43fd-8aa7-01393d9a01d1-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.379675 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c9c626a-52a7-466e-b744-f4bd79112fd9" volumeName="kubernetes.io/secret/6c9c626a-52a7-466e-b744-f4bd79112fd9-original-pull-secret" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.379689 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1" volumeName="kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-config-volume" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.379700 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1" volumeName="kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.379712 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9f7709e-2664-452f-b70b-f4649675781e" volumeName="kubernetes.io/empty-dir/d9f7709e-2664-452f-b70b-f4649675781e-tmp-dir" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.379724 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74930384-19fd-4c0b-b38f-b2f895f7e6e8" volumeName="kubernetes.io/projected/74930384-19fd-4c0b-b38f-b2f895f7e6e8-kube-api-access-vpccd" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.379735 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="166e5f35-0987-4eef-917a-a4772acba82f" volumeName="kubernetes.io/secret/166e5f35-0987-4eef-917a-a4772acba82f-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.379748 5748 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979" volumeName="kubernetes.io/empty-dir/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979-node-exporter-textfile" seLinuxMountContext="" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.379760 5748 reconstruct.go:97] "Volume reconstruction finished" Oct 30 21:23:36.379736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.379768 5748 reconciler.go:29] "Reconciler: start to sync state" Oct 30 21:23:36.401982 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.401943 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j4v8r_3df036ff-1c00-4ce2-9e86-8acbeedf6c82/ovn-acl-logging/0.log" Oct 30 21:23:36.402922 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.402874 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j4v8r_3df036ff-1c00-4ce2-9e86-8acbeedf6c82/ovn-acl-logging/0.log" Oct 30 21:23:36.407518 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.407484 5748 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 21:23:36.408886 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.408865 5748 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 21:23:36.408999 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.408890 5748 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 21:23:36.408999 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.408929 5748 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 21:23:36.408999 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:23:36.408970 5748 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 21:23:36.443630 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.443540 5748 generic.go:356] "Generic (PLEG): container finished" podID="6a9841f4-72b0-43fd-8aa7-01393d9a01d1" containerID="52ba5ea380e86ac751943c6a3263da3899051502cbcc1801c3280c089f73417e" exitCode=0 Oct 30 21:23:36.443630 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.443579 5748 generic.go:356] "Generic (PLEG): container finished" podID="6a9841f4-72b0-43fd-8aa7-01393d9a01d1" containerID="3ed99be6f4175ed8899e22f63d55c111048760666e8a2b642b31d447c0ac133f" exitCode=0 Oct 30 21:23:36.443630 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.443589 5748 generic.go:356] "Generic (PLEG): container finished" podID="6a9841f4-72b0-43fd-8aa7-01393d9a01d1" containerID="4196b9d7d2ebc1146ac9a6bd61da349478ea97330dd69555a2d48e2f671b3947" exitCode=0 Oct 30 21:23:36.443630 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.443595 5748 generic.go:356] "Generic (PLEG): container finished" podID="6a9841f4-72b0-43fd-8aa7-01393d9a01d1" containerID="d16531bebe49eb52d64188fb04118b94477466fab1cc36a225fa9c904dbb5107" exitCode=0 Oct 30 21:23:36.443630 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.443601 5748 generic.go:356] "Generic (PLEG): container finished" podID="6a9841f4-72b0-43fd-8aa7-01393d9a01d1" containerID="f73778c6a3166636aaac439843cd6d9cfde39e237e9572eb350e6225c8a999f0" exitCode=0 Oct 30 21:23:36.443630 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.443606 5748 generic.go:356] "Generic (PLEG): container finished" podID="6a9841f4-72b0-43fd-8aa7-01393d9a01d1" containerID="77827c8c81a7fd96b753fc502c45fbbf24ad8d7ce6cf8e753275b4f067d42d90" exitCode=0 Oct 30 21:23:36.449547 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.449523 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j4v8r_3df036ff-1c00-4ce2-9e86-8acbeedf6c82/ovn-acl-logging/0.log" Oct 30 21:23:36.449977 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.449944 5748 generic.go:356] "Generic (PLEG): container finished" podID="3df036ff-1c00-4ce2-9e86-8acbeedf6c82" containerID="ef98c47b4672ee0779d14d54ce9aed56e9bd752bc87e11fa1af5e8abd943fc62" exitCode=1 Oct 30 21:23:36.456673 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.456618 5748 generic.go:356] "Generic (PLEG): container finished" podID="1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979" containerID="c98bd75a6f280f92242c8c79084641d1ae66ee4a29442f1d2a97ff7ab1708a01" exitCode=0 Oct 30 21:23:36.469506 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.469474 5748 generic.go:356] "Generic (PLEG): container finished" podID="ddfcdf71a39fda8bf039a6129472bc5e" containerID="b66a7a5f218ecb9b19b19ec59bd98b65015cc0de8d93e9717506f216ddee69b9" exitCode=0 Oct 30 21:23:36.471491 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:23:36.471459 5748 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-61.us-west-1.compute.internal\" not found" Oct 30 21:23:36.481439 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.481411 5748 generic.go:356] "Generic (PLEG): container finished" podID="d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1" containerID="ffef4be78f8c37e05b78d506ffd53a0b58cd89c523b3cab54930fca37f7ccc6a" exitCode=0 Oct 30 21:23:36.502061 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.502035 5748 manager.go:324] Recovery completed Oct 30 21:23:36.509949 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:23:36.509918 5748 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 21:23:36.523918 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.523863 5748 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:23:36.526310 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.526283 5748 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-61.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:23:36.526437 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.526328 5748 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-61.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:23:36.526437 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.526346 5748 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-61.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:23:36.529350 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.529330 5748 cpu_manager.go:226] "Starting" policy="none" Oct 30 21:23:36.529350 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.529344 5748 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 21:23:36.529350 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.529349 5748 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:23:36.529562 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.529362 5748 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 21:23:36.529562 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.529491 5748 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 21:23:36.529562 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.529501 5748 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 21:23:36.529562 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.529517 5748 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 21:23:36.529562 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.529523 5748 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 21:23:36.529562 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.529529 5748 policy_none.go:50] "Start" Oct 30 21:23:36.529562 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.529559 5748 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 21:23:36.529750 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.529569 5748 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 21:23:36.529750 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.529693 5748 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 21:23:36.529750 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.529700 5748 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 21:23:36.529750 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.529705 5748 policy_none.go:44] "Start" Oct 30 21:23:36.530430 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.530412 5748 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-61.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:23:36.530516 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.530447 5748 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-61.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:23:36.530516 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.530469 5748 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-61.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:23:36.534616 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.534589 5748 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 21:23:36.537371 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.537355 5748 manager.go:341] "Starting Device Plugin manager" Oct 30 21:23:36.537449 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:23:36.537382 5748 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 21:23:36.537449 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.537391 5748 server.go:87] "Starting device plugin registration server" Oct 30 21:23:36.537746 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.537733 5748 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 21:23:36.538584 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.537750 5748 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 21:23:36.538584 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.538550 5748 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 21:23:36.538704 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.538654 5748 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 21:23:36.538704 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.538694 5748 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 21:23:36.538704 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.538701 5748 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 21:23:36.538827 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.538751 5748 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:23:36.538682598Z","UUID":"6da51e60-778d-4b92-8b15-70d18fcd3206","Handler":null,"Name":"","Endpoint":""} Oct 30 21:23:36.539993 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:23:36.539883 5748 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 21:23:36.539993 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:23:36.539936 5748 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-61.us-west-1.compute.internal\" not found" Oct 30 21:23:36.540109 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.540073 5748 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 21:23:36.540109 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.540096 5748 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 21:23:36.561832 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.561797 5748 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:23:36.562036 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.562010 5748 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 21:23:36.571385 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.571355 5748 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:23:36.571752 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.571736 5748 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:23:36.639502 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.639468 5748 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:23:36.640720 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.640700 5748 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-61.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:23:36.640815 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.640738 5748 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-61.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:23:36.640815 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.640749 5748 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-61.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:23:36.640815 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.640800 5748 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-61.us-west-1.compute.internal" Oct 30 21:23:36.662533 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.662506 5748 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:23:36.710438 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.710309 5748 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-61.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-61.us-west-1.compute.internal"] Oct 30 21:23:36.711094 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.711077 5748 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-61.us-west-1.compute.internal" Oct 30 21:23:36.711167 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.711083 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-61.us-west-1.compute.internal" event={"ID":"ddfcdf71a39fda8bf039a6129472bc5e","Type":"ContainerStarted","Data":"45745fa7e9c1e170441829c6d7712210c0a40666f8edadf521bffbbb608a90d9"} Oct 30 21:23:36.711167 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.711125 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-61.us-west-1.compute.internal" event={"ID":"ddfcdf71a39fda8bf039a6129472bc5e","Type":"ContainerDied","Data":"b66a7a5f218ecb9b19b19ec59bd98b65015cc0de8d93e9717506f216ddee69b9"} Oct 30 21:23:36.711167 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.711139 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-61.us-west-1.compute.internal" event={"ID":"ddfcdf71a39fda8bf039a6129472bc5e","Type":"ContainerStarted","Data":"2e8ccef945c9b2ab26da19f76a836a73756872b7554e0a67985accacdcb37fe9"} Oct 30 21:23:36.711167 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.711159 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-61.us-west-1.compute.internal" event={"ID":"f05411f241c17e725b228582bf16173f","Type":"ContainerStarted","Data":"2e14ed1439248491146a102d5629ecd25c0f853e53d5c09e5da28c8e46eb2176"} Oct 30 21:23:36.711290 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.711170 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-61.us-west-1.compute.internal" event={"ID":"f05411f241c17e725b228582bf16173f","Type":"ContainerStarted","Data":"be8bf0429e5f966f001d3467b942c1f2c894b8e25c67bdba0df0bce3b28e9b59"} Oct 30 21:23:36.711322 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.711300 5748 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-61.us-west-1.compute.internal" Oct 30 21:23:36.752055 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.752024 5748 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-143-61.us-west-1.compute.internal" Oct 30 21:23:36.752196 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.752105 5748 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-61.us-west-1.compute.internal" Oct 30 21:23:36.775079 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.775051 5748 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:23:36.775200 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:23:36.775097 5748 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-61.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-61.us-west-1.compute.internal" Oct 30 21:23:36.778267 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.778251 5748 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:23:36.778318 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:23:36.778288 5748 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-61.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-61.us-west-1.compute.internal" Oct 30 21:23:36.863359 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.863321 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ddfcdf71a39fda8bf039a6129472bc5e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-61.us-west-1.compute.internal\" (UID: \"ddfcdf71a39fda8bf039a6129472bc5e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-61.us-west-1.compute.internal" Oct 30 21:23:36.863359 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.863359 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ddfcdf71a39fda8bf039a6129472bc5e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-61.us-west-1.compute.internal\" (UID: \"ddfcdf71a39fda8bf039a6129472bc5e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-61.us-west-1.compute.internal" Oct 30 21:23:36.863564 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.863381 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f05411f241c17e725b228582bf16173f-config\") pod \"kube-apiserver-proxy-ip-10-0-143-61.us-west-1.compute.internal\" (UID: \"f05411f241c17e725b228582bf16173f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-61.us-west-1.compute.internal" Oct 30 21:23:36.964219 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.964132 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ddfcdf71a39fda8bf039a6129472bc5e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-61.us-west-1.compute.internal\" (UID: \"ddfcdf71a39fda8bf039a6129472bc5e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-61.us-west-1.compute.internal" Oct 30 21:23:36.964219 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.964158 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ddfcdf71a39fda8bf039a6129472bc5e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-61.us-west-1.compute.internal\" (UID: \"ddfcdf71a39fda8bf039a6129472bc5e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-61.us-west-1.compute.internal" Oct 30 21:23:36.964383 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.964230 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ddfcdf71a39fda8bf039a6129472bc5e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-61.us-west-1.compute.internal\" (UID: \"ddfcdf71a39fda8bf039a6129472bc5e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-61.us-west-1.compute.internal" Oct 30 21:23:36.964383 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.964257 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f05411f241c17e725b228582bf16173f-config\") pod \"kube-apiserver-proxy-ip-10-0-143-61.us-west-1.compute.internal\" (UID: \"f05411f241c17e725b228582bf16173f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-61.us-west-1.compute.internal" Oct 30 21:23:36.964383 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.964284 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ddfcdf71a39fda8bf039a6129472bc5e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-61.us-west-1.compute.internal\" (UID: \"ddfcdf71a39fda8bf039a6129472bc5e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-61.us-west-1.compute.internal" Oct 30 21:23:36.964383 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:36.964299 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f05411f241c17e725b228582bf16173f-config\") pod \"kube-apiserver-proxy-ip-10-0-143-61.us-west-1.compute.internal\" (UID: \"f05411f241c17e725b228582bf16173f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-61.us-west-1.compute.internal" Oct 30 21:23:37.340035 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.339736 5748 apiserver.go:52] "Watching apiserver" Oct 30 21:23:37.450563 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.450524 5748 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 21:23:37.451326 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.451300 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vfrjv","kube-system/kube-apiserver-proxy-ip-10-0-143-61.us-west-1.compute.internal","openshift-ingress-canary/ingress-canary-zb9t2","openshift-dns/dns-default-nrm49","openshift-multus/multus-n2sfk","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvfbv","openshift-image-registry/node-ca-8vqf9","openshift-monitoring/kube-state-metrics-5dc8d7d89b-kqxdq","openshift-console/downloads-6dd789c8b7-rq2vl","openshift-multus/multus-additional-cni-plugins-jtrcb","openshift-monitoring/prometheus-operator-9cd89b74c-fglqz","kube-system/global-pull-secret-syncer-6nncz","openshift-monitoring/metrics-server-5dcb975b97-crpzf","openshift-ovn-kubernetes/ovnkube-node-j4v8r","openshift-monitoring/node-exporter-rlt89","openshift-monitoring/alertmanager-main-0","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-61.us-west-1.compute.internal","openshift-network-diagnostics/network-check-target-wv464","openshift-cluster-node-tuning-operator/tuned-dpr6c","openshift-monitoring/monitoring-plugin-77f4f76594-2r74x","openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7qc5s","openshift-monitoring/openshift-state-metrics-7bd7489957-jfpf9","kube-system/konnectivity-agent-75hkd","openshift-network-operator/iptables-alerter-mxfq6","openshift-multus/network-metrics-daemon-lz8cq","openshift-dns/node-resolver-8rbsm","openshift-monitoring/thanos-querier-76bbc756f6-s5wtw","openshift-monitoring/prometheus-k8s-0","openshift-console/console-7fbd54bb44-d7zl5","openshift-monitoring/telemeter-client-7c58f6d48-ls9th"] Oct 30 21:23:37.480617 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.480584 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:37.500118 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.500090 5748 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-61.us-west-1.compute.internal" Oct 30 21:23:37.500269 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.500208 5748 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-61.us-west-1.compute.internal" Oct 30 21:23:37.507067 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.507040 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 21:23:37.507067 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.507063 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-v6bcb\"" Oct 30 21:23:37.507251 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.507040 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-gmgqb\"" Oct 30 21:23:37.508104 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.508085 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:23:37.508223 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.508190 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 21:23:37.508223 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.508201 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 21:23:37.508223 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.508212 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 21:23:37.508223 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.508216 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 21:23:37.508391 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.508231 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 21:23:37.508391 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.508260 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 21:23:37.508391 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.508284 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 21:23:37.516280 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.516255 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 21:23:37.516436 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.516414 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:23:37.516501 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.516489 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 21:23:37.516536 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.516522 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 21:23:37.516568 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.516555 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 21:23:37.516693 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.516682 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 21:23:37.516725 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.516717 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 21:23:37.516815 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.516803 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 21:23:37.516856 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.516842 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-vm9q9\"" Oct 30 21:23:37.516910 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.516884 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 21:23:37.516942 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.516888 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-mbjr2\"" Oct 30 21:23:37.516979 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.516956 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-ntkmx\"" Oct 30 21:23:37.517749 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.517737 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 21:23:37.519326 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.519311 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-gkfxf\"" Oct 30 21:23:37.519326 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.519319 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7lclr\"" Oct 30 21:23:37.519385 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.519311 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-7rrvk\"" Oct 30 21:23:37.519471 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.519460 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-rf4nn\"" Oct 30 21:23:37.520650 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.520638 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 21:23:37.520747 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.520730 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xqzlq\"" Oct 30 21:23:37.520810 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.520796 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nr2gm\"" Oct 30 21:23:37.520880 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.520870 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-jj5qw\"" Oct 30 21:23:37.521472 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.521456 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 21:23:37.521472 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.521464 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 21:23:37.521585 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.521544 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 21:23:37.521585 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.521567 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 21:23:37.521663 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.521598 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 21:23:37.521663 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.521615 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 21:23:37.521663 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.521620 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 21:23:37.521763 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.521656 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 21:23:37.521763 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.521698 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 21:23:37.521816 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.521799 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 21:23:37.563059 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.563036 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 21:23:37.565132 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.565107 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 21:23:37.565132 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.565107 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 21:23:37.565265 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.565189 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 21:23:37.565361 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.565333 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 21:23:37.565361 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.565354 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 21:23:37.565505 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.565378 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 21:23:37.565571 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.565556 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 21:23:37.565633 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.565616 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 21:23:37.565678 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.565638 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 21:23:37.565678 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.565656 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 21:23:37.566770 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.566752 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 21:23:37.566845 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.566791 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5nrzq\"" Oct 30 21:23:37.566845 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.566796 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 21:23:37.566845 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.566813 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-82qgx\"" Oct 30 21:23:37.567063 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.567050 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pvd8m\"" Oct 30 21:23:37.567367 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.567350 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vjpnc\"" Oct 30 21:23:37.567442 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.567358 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 21:23:37.567442 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.567380 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-bdcrp\"" Oct 30 21:23:37.567442 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.567398 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 21:23:37.567442 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.567405 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 21:23:37.567578 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.567407 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 21:23:37.568114 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.568097 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 21:23:37.568170 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.568125 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 21:23:37.568170 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.568164 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 21:23:37.568310 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.568294 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 21:23:37.568397 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.568325 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 21:23:37.568397 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.568387 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 21:23:37.568478 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.568423 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 21:23:37.568478 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.568424 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 21:23:37.568619 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.568603 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 21:23:37.568708 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.568621 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 21:23:37.568708 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.568670 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 30 21:23:37.568708 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.568687 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 21:23:37.568876 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.568711 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 21:23:37.568957 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.568876 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-4pcbl\"" Oct 30 21:23:37.569009 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.568996 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-598dq\"" Oct 30 21:23:37.569714 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.569696 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-host-cni-netd\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:23:37.569751 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.569731 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a597eced-76f6-49e8-8176-2e9332188be3-secret-thanos-querier-tls\") pod \"thanos-querier-76bbc756f6-s5wtw\" (UID: \"a597eced-76f6-49e8-8176-2e9332188be3\") " pod="openshift-monitoring/thanos-querier-76bbc756f6-s5wtw" Oct 30 21:23:37.569790 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.569749 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5c3735e3-95ac-4293-8555-38ee9ead0353-iptables-alerter-script\") pod \"iptables-alerter-mxfq6\" (UID: \"5c3735e3-95ac-4293-8555-38ee9ead0353\") " pod="openshift-network-operator/iptables-alerter-mxfq6" Oct 30 21:23:37.569790 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.569768 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/aed61d9f-eb67-44ae-a094-a12814e42559-socket-dir\") pod \"aws-ebs-csi-driver-node-lvfbv\" (UID: \"aed61d9f-eb67-44ae-a094-a12814e42559\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvfbv" Oct 30 21:23:37.569866 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.569790 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/42aedd5a-da26-4be4-8ed8-01fb25f3ee5f-crio-socket\") pod \"insights-runtime-extractor-vfrjv\" (UID: \"42aedd5a-da26-4be4-8ed8-01fb25f3ee5f\") " pod="openshift-insights/insights-runtime-extractor-vfrjv" Oct 30 21:23:37.569924 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.569862 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-etc-sysconfig\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:23:37.569924 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.569909 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:37.570057 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.570037 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-systemd-units\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:23:37.570130 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.570112 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-etc-openvswitch\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:23:37.570177 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.570151 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpccd\" (UniqueName: \"kubernetes.io/projected/74930384-19fd-4c0b-b38f-b2f895f7e6e8-kube-api-access-vpccd\") pod \"ingress-canary-zb9t2\" (UID: \"74930384-19fd-4c0b-b38f-b2f895f7e6e8\") " pod="openshift-ingress-canary/ingress-canary-zb9t2" Oct 30 21:23:37.570177 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.570166 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5c3735e3-95ac-4293-8555-38ee9ead0353-iptables-alerter-script\") pod \"iptables-alerter-mxfq6\" (UID: \"5c3735e3-95ac-4293-8555-38ee9ead0353\") " pod="openshift-network-operator/iptables-alerter-mxfq6" Oct 30 21:23:37.570245 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.570182 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6a9841f4-72b0-43fd-8aa7-01393d9a01d1-cni-binary-copy\") pod \"multus-additional-cni-plugins-jtrcb\" (UID: \"6a9841f4-72b0-43fd-8aa7-01393d9a01d1\") " pod="openshift-multus/multus-additional-cni-plugins-jtrcb" Oct 30 21:23:37.570245 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.570186 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:37.570245 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.570226 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-multus-socket-dir-parent\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:23:37.570340 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.570266 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-host-var-lib-kubelet\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:23:37.570340 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.570295 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:37.570340 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.570331 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca9f1917-07a8-4ea6-8b59-4f2f99429dfa-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-jfpf9\" (UID: \"ca9f1917-07a8-4ea6-8b59-4f2f99429dfa\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jfpf9" Oct 30 21:23:37.570455 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.570361 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6a9841f4-72b0-43fd-8aa7-01393d9a01d1-cni-binary-copy\") pod \"multus-additional-cni-plugins-jtrcb\" (UID: \"6a9841f4-72b0-43fd-8aa7-01393d9a01d1\") " pod="openshift-multus/multus-additional-cni-plugins-jtrcb" Oct 30 21:23:37.570455 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.570425 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c2b5bbd1-4e37-4d49-9e28-bdb3c3da309f-config-volume\") pod \"dns-default-nrm49\" (UID: \"c2b5bbd1-4e37-4d49-9e28-bdb3c3da309f\") " pod="openshift-dns/dns-default-nrm49" Oct 30 21:23:37.570559 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.570506 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:37.570559 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.570537 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979-node-exporter-tls\") pod \"node-exporter-rlt89\" (UID: \"1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979\") " pod="openshift-monitoring/node-exporter-rlt89" Oct 30 21:23:37.570656 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.570565 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-etc-modprobe-d\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:23:37.570656 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.570592 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-var-lib-kubelet\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:23:37.570656 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.570615 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca9f1917-07a8-4ea6-8b59-4f2f99429dfa-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-jfpf9\" (UID: \"ca9f1917-07a8-4ea6-8b59-4f2f99429dfa\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jfpf9" Oct 30 21:23:37.570656 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.570619 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4v76\" (UniqueName: \"kubernetes.io/projected/8ea6188f-9e30-4ab0-b5b3-c99647377e00-kube-api-access-b4v76\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:23:37.570782 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.570654 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c2b5bbd1-4e37-4d49-9e28-bdb3c3da309f-config-volume\") pod \"dns-default-nrm49\" (UID: \"c2b5bbd1-4e37-4d49-9e28-bdb3c3da309f\") " pod="openshift-dns/dns-default-nrm49" Oct 30 21:23:37.570782 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.570668 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-multus-cni-dir\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:23:37.570782 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.570715 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-host-var-lib-cni-bin\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:23:37.570782 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.570747 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83a93696-9f3b-472d-9fc8-92350746df75-metrics-certs\") pod \"network-metrics-daemon-lz8cq\" (UID: \"83a93696-9f3b-472d-9fc8-92350746df75\") " pod="openshift-multus/network-metrics-daemon-lz8cq" Oct 30 21:23:37.570782 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.570747 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979-node-exporter-tls\") pod \"node-exporter-rlt89\" (UID: \"1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979\") " pod="openshift-monitoring/node-exporter-rlt89" Oct 30 21:23:37.570782 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.570777 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/166e5f35-0987-4eef-917a-a4772acba82f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-kqxdq\" (UID: \"166e5f35-0987-4eef-917a-a4772acba82f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-kqxdq" Oct 30 21:23:37.571110 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.570868 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:37.571110 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.570933 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-env-overrides\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:23:37.571110 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.570961 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c3735e3-95ac-4293-8555-38ee9ead0353-host-slash\") pod \"iptables-alerter-mxfq6\" (UID: \"5c3735e3-95ac-4293-8555-38ee9ead0353\") " pod="openshift-network-operator/iptables-alerter-mxfq6" Oct 30 21:23:37.571110 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.570963 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83a93696-9f3b-472d-9fc8-92350746df75-metrics-certs\") pod \"network-metrics-daemon-lz8cq\" (UID: \"83a93696-9f3b-472d-9fc8-92350746df75\") " pod="openshift-multus/network-metrics-daemon-lz8cq" Oct 30 21:23:37.571110 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.570980 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/454a9788-f96e-453d-a83c-63c550c7d96d-konnectivity-ca\") pod \"konnectivity-agent-75hkd\" (UID: \"454a9788-f96e-453d-a83c-63c550c7d96d\") " pod="kube-system/konnectivity-agent-75hkd" Oct 30 21:23:37.571110 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.571037 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/aed61d9f-eb67-44ae-a094-a12814e42559-etc-selinux\") pod \"aws-ebs-csi-driver-node-lvfbv\" (UID: \"aed61d9f-eb67-44ae-a094-a12814e42559\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvfbv" Oct 30 21:23:37.571110 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.571041 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/166e5f35-0987-4eef-917a-a4772acba82f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-kqxdq\" (UID: \"166e5f35-0987-4eef-917a-a4772acba82f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-kqxdq" Oct 30 21:23:37.571410 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.571121 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6a9841f4-72b0-43fd-8aa7-01393d9a01d1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jtrcb\" (UID: \"6a9841f4-72b0-43fd-8aa7-01393d9a01d1\") " pod="openshift-multus/multus-additional-cni-plugins-jtrcb" Oct 30 21:23:37.571410 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.571117 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-env-overrides\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:23:37.571410 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.571177 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r5f6\" (UniqueName: \"kubernetes.io/projected/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-kube-api-access-6r5f6\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:23:37.571410 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.571226 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2b5bbd1-4e37-4d49-9e28-bdb3c3da309f-tmp-dir\") pod \"dns-default-nrm49\" (UID: \"c2b5bbd1-4e37-4d49-9e28-bdb3c3da309f\") " pod="openshift-dns/dns-default-nrm49" Oct 30 21:23:37.571410 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.571262 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km8jp\" (UniqueName: \"kubernetes.io/projected/c2b5bbd1-4e37-4d49-9e28-bdb3c3da309f-kube-api-access-km8jp\") pod \"dns-default-nrm49\" (UID: \"c2b5bbd1-4e37-4d49-9e28-bdb3c3da309f\") " pod="openshift-dns/dns-default-nrm49" Oct 30 21:23:37.571410 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.571304 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-run-openvswitch\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:23:37.571410 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.571322 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/454a9788-f96e-453d-a83c-63c550c7d96d-konnectivity-ca\") pod \"konnectivity-agent-75hkd\" (UID: \"454a9788-f96e-453d-a83c-63c550c7d96d\") " pod="kube-system/konnectivity-agent-75hkd" Oct 30 21:23:37.571410 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.571335 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/aed61d9f-eb67-44ae-a094-a12814e42559-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lvfbv\" (UID: \"aed61d9f-eb67-44ae-a094-a12814e42559\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvfbv" Oct 30 21:23:37.571410 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.571363 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6a9841f4-72b0-43fd-8aa7-01393d9a01d1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jtrcb\" (UID: \"6a9841f4-72b0-43fd-8aa7-01393d9a01d1\") " pod="openshift-multus/multus-additional-cni-plugins-jtrcb" Oct 30 21:23:37.571410 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.571385 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2b5bbd1-4e37-4d49-9e28-bdb3c3da309f-tmp-dir\") pod \"dns-default-nrm49\" (UID: \"c2b5bbd1-4e37-4d49-9e28-bdb3c3da309f\") " pod="openshift-dns/dns-default-nrm49" Oct 30 21:23:37.571689 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.571433 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6mc5\" (UniqueName: \"kubernetes.io/projected/6a9841f4-72b0-43fd-8aa7-01393d9a01d1-kube-api-access-g6mc5\") pod \"multus-additional-cni-plugins-jtrcb\" (UID: \"6a9841f4-72b0-43fd-8aa7-01393d9a01d1\") " pod="openshift-multus/multus-additional-cni-plugins-jtrcb" Oct 30 21:23:37.571689 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.571474 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpbmf\" (UniqueName: \"kubernetes.io/projected/2496f073-7bf0-48af-b091-be6b964117ca-kube-api-access-vpbmf\") pod \"network-check-target-wv464\" (UID: \"2496f073-7bf0-48af-b091-be6b964117ca\") " pod="openshift-network-diagnostics/network-check-target-wv464" Oct 30 21:23:37.571689 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.571526 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-host\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:23:37.571689 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.571547 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/166e5f35-0987-4eef-917a-a4772acba82f-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-kqxdq\" (UID: \"166e5f35-0987-4eef-917a-a4772acba82f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-kqxdq" Oct 30 21:23:37.571689 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.571565 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-config-out\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:37.571689 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.571581 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/74930384-19fd-4c0b-b38f-b2f895f7e6e8-cert\") pod \"ingress-canary-zb9t2\" (UID: \"74930384-19fd-4c0b-b38f-b2f895f7e6e8\") " pod="openshift-ingress-canary/ingress-canary-zb9t2" Oct 30 21:23:37.571689 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.571597 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/42aedd5a-da26-4be4-8ed8-01fb25f3ee5f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vfrjv\" (UID: \"42aedd5a-da26-4be4-8ed8-01fb25f3ee5f\") " pod="openshift-insights/insights-runtime-extractor-vfrjv" Oct 30 21:23:37.571689 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.571613 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6a9841f4-72b0-43fd-8aa7-01393d9a01d1-cnibin\") pod \"multus-additional-cni-plugins-jtrcb\" (UID: \"6a9841f4-72b0-43fd-8aa7-01393d9a01d1\") " pod="openshift-multus/multus-additional-cni-plugins-jtrcb" Oct 30 21:23:37.571689 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.571628 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-etc-systemd\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:23:37.571689 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.571644 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-system-cni-dir\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:23:37.571689 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.571659 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-config-volume\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:37.571689 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.571673 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:37.571689 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.571691 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-run-systemd\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:23:37.572194 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.571709 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a9841f4-72b0-43fd-8aa7-01393d9a01d1-system-cni-dir\") pod \"multus-additional-cni-plugins-jtrcb\" (UID: \"6a9841f4-72b0-43fd-8aa7-01393d9a01d1\") " pod="openshift-multus/multus-additional-cni-plugins-jtrcb" Oct 30 21:23:37.572194 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.571725 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs2w6\" (UniqueName: \"kubernetes.io/projected/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979-kube-api-access-hs2w6\") pod \"node-exporter-rlt89\" (UID: \"1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979\") " pod="openshift-monitoring/node-exporter-rlt89" Oct 30 21:23:37.572194 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.571739 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-multus-conf-dir\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:23:37.572194 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.571754 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/166e5f35-0987-4eef-917a-a4772acba82f-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-kqxdq\" (UID: \"166e5f35-0987-4eef-917a-a4772acba82f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-kqxdq" Oct 30 21:23:37.572194 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.571771 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-host-slash\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:23:37.572194 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.571779 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/74930384-19fd-4c0b-b38f-b2f895f7e6e8-cert\") pod \"ingress-canary-zb9t2\" (UID: \"74930384-19fd-4c0b-b38f-b2f895f7e6e8\") " pod="openshift-ingress-canary/ingress-canary-zb9t2" Oct 30 21:23:37.572194 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.571786 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-node-log\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:23:37.572194 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.571796 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/166e5f35-0987-4eef-917a-a4772acba82f-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-kqxdq\" (UID: \"166e5f35-0987-4eef-917a-a4772acba82f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-kqxdq" Oct 30 21:23:37.572194 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.571796 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-config-out\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:37.572194 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.571823 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-host-run-ovn-kubernetes\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:23:37.572194 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.571875 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4dgx\" (UniqueName: \"kubernetes.io/projected/aed61d9f-eb67-44ae-a094-a12814e42559-kube-api-access-c4dgx\") pod \"aws-ebs-csi-driver-node-lvfbv\" (UID: \"aed61d9f-eb67-44ae-a094-a12814e42559\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvfbv" Oct 30 21:23:37.572194 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.571886 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/166e5f35-0987-4eef-917a-a4772acba82f-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-kqxdq\" (UID: \"166e5f35-0987-4eef-917a-a4772acba82f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-kqxdq" Oct 30 21:23:37.572194 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.572007 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-host-run-multus-certs\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:23:37.572194 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.572042 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw8lt\" (UniqueName: \"kubernetes.io/projected/42aedd5a-da26-4be4-8ed8-01fb25f3ee5f-kube-api-access-kw8lt\") pod \"insights-runtime-extractor-vfrjv\" (UID: \"42aedd5a-da26-4be4-8ed8-01fb25f3ee5f\") " pod="openshift-insights/insights-runtime-extractor-vfrjv" Oct 30 21:23:37.572194 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.572067 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8gxg\" (UniqueName: \"kubernetes.io/projected/83a93696-9f3b-472d-9fc8-92350746df75-kube-api-access-b8gxg\") pod \"network-metrics-daemon-lz8cq\" (UID: \"83a93696-9f3b-472d-9fc8-92350746df75\") " pod="openshift-multus/network-metrics-daemon-lz8cq" Oct 30 21:23:37.572194 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.572084 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6c9c626a-52a7-466e-b744-f4bd79112fd9-original-pull-secret\") pod \"global-pull-secret-syncer-6nncz\" (UID: \"6c9c626a-52a7-466e-b744-f4bd79112fd9\") " pod="kube-system/global-pull-secret-syncer-6nncz" Oct 30 21:23:37.572194 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.572093 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:37.572194 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.572135 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6a9841f4-72b0-43fd-8aa7-01393d9a01d1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jtrcb\" (UID: \"6a9841f4-72b0-43fd-8aa7-01393d9a01d1\") " pod="openshift-multus/multus-additional-cni-plugins-jtrcb" Oct 30 21:23:37.572194 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.572174 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d9f7709e-2664-452f-b70b-f4649675781e-hosts-file\") pod \"node-resolver-8rbsm\" (UID: \"d9f7709e-2664-452f-b70b-f4649675781e\") " pod="openshift-dns/node-resolver-8rbsm" Oct 30 21:23:37.572804 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.572218 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/42aedd5a-da26-4be4-8ed8-01fb25f3ee5f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vfrjv\" (UID: \"42aedd5a-da26-4be4-8ed8-01fb25f3ee5f\") " pod="openshift-insights/insights-runtime-extractor-vfrjv" Oct 30 21:23:37.572804 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.572291 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9f7709e-2664-452f-b70b-f4649675781e-tmp-dir\") pod \"node-resolver-8rbsm\" (UID: \"d9f7709e-2664-452f-b70b-f4649675781e\") " pod="openshift-dns/node-resolver-8rbsm" Oct 30 21:23:37.572804 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.572308 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6a9841f4-72b0-43fd-8aa7-01393d9a01d1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jtrcb\" (UID: \"6a9841f4-72b0-43fd-8aa7-01393d9a01d1\") " pod="openshift-multus/multus-additional-cni-plugins-jtrcb" Oct 30 21:23:37.572804 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.572427 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a597eced-76f6-49e8-8176-2e9332188be3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-76bbc756f6-s5wtw\" (UID: \"a597eced-76f6-49e8-8176-2e9332188be3\") " pod="openshift-monitoring/thanos-querier-76bbc756f6-s5wtw" Oct 30 21:23:37.572804 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.572472 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9f7709e-2664-452f-b70b-f4649675781e-tmp-dir\") pod \"node-resolver-8rbsm\" (UID: \"d9f7709e-2664-452f-b70b-f4649675781e\") " pod="openshift-dns/node-resolver-8rbsm" Oct 30 21:23:37.572804 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.572472 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/454a9788-f96e-453d-a83c-63c550c7d96d-agent-certs\") pod \"konnectivity-agent-75hkd\" (UID: \"454a9788-f96e-453d-a83c-63c550c7d96d\") " pod="kube-system/konnectivity-agent-75hkd" Oct 30 21:23:37.572804 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.572522 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a597eced-76f6-49e8-8176-2e9332188be3-tmp\") pod \"thanos-querier-76bbc756f6-s5wtw\" (UID: \"a597eced-76f6-49e8-8176-2e9332188be3\") " pod="openshift-monitoring/thanos-querier-76bbc756f6-s5wtw" Oct 30 21:23:37.572804 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.572566 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c2b5bbd1-4e37-4d49-9e28-bdb3c3da309f-metrics-tls\") pod \"dns-default-nrm49\" (UID: \"c2b5bbd1-4e37-4d49-9e28-bdb3c3da309f\") " pod="openshift-dns/dns-default-nrm49" Oct 30 21:23:37.572804 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.572592 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979-sys\") pod \"node-exporter-rlt89\" (UID: \"1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979\") " pod="openshift-monitoring/node-exporter-rlt89" Oct 30 21:23:37.572804 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.572617 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:37.572804 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.572633 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a597eced-76f6-49e8-8176-2e9332188be3-tmp\") pod \"thanos-querier-76bbc756f6-s5wtw\" (UID: \"a597eced-76f6-49e8-8176-2e9332188be3\") " pod="openshift-monitoring/thanos-querier-76bbc756f6-s5wtw" Oct 30 21:23:37.572804 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.572642 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-log-socket\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:23:37.572804 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.572656 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/454a9788-f96e-453d-a83c-63c550c7d96d-agent-certs\") pod \"konnectivity-agent-75hkd\" (UID: \"454a9788-f96e-453d-a83c-63c550c7d96d\") " pod="kube-system/konnectivity-agent-75hkd" Oct 30 21:23:37.572804 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.572729 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bghfr\" (UniqueName: \"kubernetes.io/projected/ca9f1917-07a8-4ea6-8b59-4f2f99429dfa-kube-api-access-bghfr\") pod \"openshift-state-metrics-7bd7489957-jfpf9\" (UID: \"ca9f1917-07a8-4ea6-8b59-4f2f99429dfa\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jfpf9" Oct 30 21:23:37.572804 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.572767 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a9ca84f8-a6d5-45f4-9137-799e05b12c11-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-fglqz\" (UID: \"a9ca84f8-a6d5-45f4-9137-799e05b12c11\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fglqz" Oct 30 21:23:37.573445 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.572817 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw9tn\" (UniqueName: \"kubernetes.io/projected/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-kube-api-access-sw9tn\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:37.573445 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.572824 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:37.573445 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.572946 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-ovnkube-script-lib\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:23:37.573445 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.572996 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c2b5bbd1-4e37-4d49-9e28-bdb3c3da309f-metrics-tls\") pod \"dns-default-nrm49\" (UID: \"c2b5bbd1-4e37-4d49-9e28-bdb3c3da309f\") " pod="openshift-dns/dns-default-nrm49" Oct 30 21:23:37.573445 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.573006 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979-metrics-client-ca\") pod \"node-exporter-rlt89\" (UID: \"1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979\") " pod="openshift-monitoring/node-exporter-rlt89" Oct 30 21:23:37.573445 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.573052 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-ovn-node-metrics-cert\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:23:37.573445 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.573082 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b282542e-5b91-430d-bd5e-03af2e875952-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-7qc5s\" (UID: \"b282542e-5b91-430d-bd5e-03af2e875952\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7qc5s" Oct 30 21:23:37.573445 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.573149 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ca9f1917-07a8-4ea6-8b59-4f2f99429dfa-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-jfpf9\" (UID: \"ca9f1917-07a8-4ea6-8b59-4f2f99429dfa\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jfpf9" Oct 30 21:23:37.573445 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.573213 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-ovnkube-script-lib\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:23:37.573445 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.573212 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979-metrics-client-ca\") pod \"node-exporter-rlt89\" (UID: \"1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979\") " pod="openshift-monitoring/node-exporter-rlt89" Oct 30 21:23:37.573445 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.573244 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979-root\") pod \"node-exporter-rlt89\" (UID: \"1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979\") " pod="openshift-monitoring/node-exporter-rlt89" Oct 30 21:23:37.573445 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.573280 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2dbd490f-d94c-4f12-9106-a12117deda9d-tmp\") pod \"downloads-6dd789c8b7-rq2vl\" (UID: \"2dbd490f-d94c-4f12-9106-a12117deda9d\") " pod="openshift-console/downloads-6dd789c8b7-rq2vl" Oct 30 21:23:37.573445 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.573314 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979-node-exporter-textfile\") pod \"node-exporter-rlt89\" (UID: \"1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979\") " pod="openshift-monitoring/node-exporter-rlt89" Oct 30 21:23:37.573445 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.573331 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-etc-kubernetes\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:23:37.573445 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.573351 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-etc-kubernetes\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:23:37.573445 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.573368 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-web-config\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:37.573445 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.573384 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:37.573445 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.573388 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979-node-exporter-textfile\") pod \"node-exporter-rlt89\" (UID: \"1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979\") " pod="openshift-monitoring/node-exporter-rlt89" Oct 30 21:23:37.573445 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.573388 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2dbd490f-d94c-4f12-9106-a12117deda9d-tmp\") pod \"downloads-6dd789c8b7-rq2vl\" (UID: \"2dbd490f-d94c-4f12-9106-a12117deda9d\") " pod="openshift-console/downloads-6dd789c8b7-rq2vl" Oct 30 21:23:37.573445 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.573421 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:37.573445 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.573445 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979-node-exporter-accelerators-collector-config\") pod \"node-exporter-rlt89\" (UID: \"1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979\") " pod="openshift-monitoring/node-exporter-rlt89" Oct 30 21:23:37.574069 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.573477 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ca9f1917-07a8-4ea6-8b59-4f2f99429dfa-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-jfpf9\" (UID: \"ca9f1917-07a8-4ea6-8b59-4f2f99429dfa\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jfpf9" Oct 30 21:23:37.574069 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.573495 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-ovn-node-metrics-cert\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:23:37.574069 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.573512 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-host-kubelet\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:23:37.574069 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.573536 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/42aedd5a-da26-4be4-8ed8-01fb25f3ee5f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vfrjv\" (UID: \"42aedd5a-da26-4be4-8ed8-01fb25f3ee5f\") " pod="openshift-insights/insights-runtime-extractor-vfrjv" Oct 30 21:23:37.574069 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.573671 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979-node-exporter-accelerators-collector-config\") pod \"node-exporter-rlt89\" (UID: \"1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979\") " pod="openshift-monitoring/node-exporter-rlt89" Oct 30 21:23:37.574069 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.573677 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-etc-sysctl-d\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:23:37.574069 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.573729 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6c9c626a-52a7-466e-b744-f4bd79112fd9-dbus\") pod \"global-pull-secret-syncer-6nncz\" (UID: \"6c9c626a-52a7-466e-b744-f4bd79112fd9\") " pod="kube-system/global-pull-secret-syncer-6nncz" Oct 30 21:23:37.574069 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.573761 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:37.574069 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.573797 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-var-lib-openvswitch\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:23:37.574069 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.573839 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rlt89\" (UID: \"1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979\") " pod="openshift-monitoring/node-exporter-rlt89" Oct 30 21:23:37.574069 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.573864 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3cabcccd-6b1e-4089-a3a6-275558de2bfb-cni-binary-copy\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:23:37.574069 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.573912 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3cabcccd-6b1e-4089-a3a6-275558de2bfb-multus-daemon-config\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:23:37.574069 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.573940 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2xzl\" (UniqueName: \"kubernetes.io/projected/2dbd490f-d94c-4f12-9106-a12117deda9d-kube-api-access-d2xzl\") pod \"downloads-6dd789c8b7-rq2vl\" (UID: \"2dbd490f-d94c-4f12-9106-a12117deda9d\") " pod="openshift-console/downloads-6dd789c8b7-rq2vl" Oct 30 21:23:37.574069 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.573967 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-run-ovn\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:23:37.574069 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.573993 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a597eced-76f6-49e8-8176-2e9332188be3-secret-grpc-tls\") pod \"thanos-querier-76bbc756f6-s5wtw\" (UID: \"a597eced-76f6-49e8-8176-2e9332188be3\") " pod="openshift-monitoring/thanos-querier-76bbc756f6-s5wtw" Oct 30 21:23:37.574069 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.574048 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3cabcccd-6b1e-4089-a3a6-275558de2bfb-cni-binary-copy\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:23:37.574715 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.574084 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3cabcccd-6b1e-4089-a3a6-275558de2bfb-multus-daemon-config\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:23:37.574715 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.574156 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7qhc\" (UniqueName: \"kubernetes.io/projected/59571898-f15b-4b8f-b043-0e5af369c39e-kube-api-access-d7qhc\") pod \"node-ca-8vqf9\" (UID: \"59571898-f15b-4b8f-b043-0e5af369c39e\") " pod="openshift-image-registry/node-ca-8vqf9" Oct 30 21:23:37.574715 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.574194 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979-node-exporter-wtmp\") pod \"node-exporter-rlt89\" (UID: \"1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979\") " pod="openshift-monitoring/node-exporter-rlt89" Oct 30 21:23:37.574715 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.574379 5748 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 21:23:37.574715 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.574400 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-lib-modules\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:23:37.574715 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.574417 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-hostroot\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:23:37.574715 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.574435 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a597eced-76f6-49e8-8176-2e9332188be3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-76bbc756f6-s5wtw\" (UID: \"a597eced-76f6-49e8-8176-2e9332188be3\") " pod="openshift-monitoring/thanos-querier-76bbc756f6-s5wtw" Oct 30 21:23:37.574715 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.574481 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a597eced-76f6-49e8-8176-2e9332188be3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-76bbc756f6-s5wtw\" (UID: \"a597eced-76f6-49e8-8176-2e9332188be3\") " pod="openshift-monitoring/thanos-querier-76bbc756f6-s5wtw" Oct 30 21:23:37.574715 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.574523 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/59571898-f15b-4b8f-b043-0e5af369c39e-host\") pod \"node-ca-8vqf9\" (UID: \"59571898-f15b-4b8f-b043-0e5af369c39e\") " pod="openshift-image-registry/node-ca-8vqf9" Oct 30 21:23:37.574715 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.574624 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9ca84f8-a6d5-45f4-9137-799e05b12c11-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-fglqz\" (UID: \"a9ca84f8-a6d5-45f4-9137-799e05b12c11\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fglqz" Oct 30 21:23:37.574715 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.574661 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:23:37.574715 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.574689 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-ovnkube-config\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:23:37.574715 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.574716 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq8kj\" (UniqueName: \"kubernetes.io/projected/5c3735e3-95ac-4293-8555-38ee9ead0353-kube-api-access-bq8kj\") pod \"iptables-alerter-mxfq6\" (UID: \"5c3735e3-95ac-4293-8555-38ee9ead0353\") " pod="openshift-network-operator/iptables-alerter-mxfq6" Oct 30 21:23:37.575332 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.574751 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-host-run-netns\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:23:37.575332 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.574780 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a9ca84f8-a6d5-45f4-9137-799e05b12c11-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-fglqz\" (UID: \"a9ca84f8-a6d5-45f4-9137-799e05b12c11\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fglqz" Oct 30 21:23:37.575332 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.574809 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/166e5f35-0987-4eef-917a-a4772acba82f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-kqxdq\" (UID: \"166e5f35-0987-4eef-917a-a4772acba82f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-kqxdq" Oct 30 21:23:37.575332 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.574820 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9ca84f8-a6d5-45f4-9137-799e05b12c11-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-fglqz\" (UID: \"a9ca84f8-a6d5-45f4-9137-799e05b12c11\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fglqz" Oct 30 21:23:37.575332 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.574854 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6a9841f4-72b0-43fd-8aa7-01393d9a01d1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jtrcb\" (UID: \"6a9841f4-72b0-43fd-8aa7-01393d9a01d1\") " pod="openshift-multus/multus-additional-cni-plugins-jtrcb" Oct 30 21:23:37.575332 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.574886 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a597eced-76f6-49e8-8176-2e9332188be3-metrics-client-ca\") pod \"thanos-querier-76bbc756f6-s5wtw\" (UID: \"a597eced-76f6-49e8-8176-2e9332188be3\") " pod="openshift-monitoring/thanos-querier-76bbc756f6-s5wtw" Oct 30 21:23:37.575332 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.574921 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-ovnkube-config\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:23:37.575332 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.574930 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/42aedd5a-da26-4be4-8ed8-01fb25f3ee5f-data-volume\") pod \"insights-runtime-extractor-vfrjv\" (UID: \"42aedd5a-da26-4be4-8ed8-01fb25f3ee5f\") " pod="openshift-insights/insights-runtime-extractor-vfrjv" Oct 30 21:23:37.575332 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.574966 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:37.575332 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.574991 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/166e5f35-0987-4eef-917a-a4772acba82f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-kqxdq\" (UID: \"166e5f35-0987-4eef-917a-a4772acba82f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-kqxdq" Oct 30 21:23:37.575332 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.574997 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/aed61d9f-eb67-44ae-a094-a12814e42559-sys-fs\") pod \"aws-ebs-csi-driver-node-lvfbv\" (UID: \"aed61d9f-eb67-44ae-a094-a12814e42559\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvfbv" Oct 30 21:23:37.575332 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.574999 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/42aedd5a-da26-4be4-8ed8-01fb25f3ee5f-data-volume\") pod \"insights-runtime-extractor-vfrjv\" (UID: \"42aedd5a-da26-4be4-8ed8-01fb25f3ee5f\") " pod="openshift-insights/insights-runtime-extractor-vfrjv" Oct 30 21:23:37.575332 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.575026 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/59571898-f15b-4b8f-b043-0e5af369c39e-serviceca\") pod \"node-ca-8vqf9\" (UID: \"59571898-f15b-4b8f-b043-0e5af369c39e\") " pod="openshift-image-registry/node-ca-8vqf9" Oct 30 21:23:37.575332 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.575074 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-host-var-lib-cni-multus\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:23:37.575332 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.575107 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/166e5f35-0987-4eef-917a-a4772acba82f-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-kqxdq\" (UID: \"166e5f35-0987-4eef-917a-a4772acba82f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-kqxdq" Oct 30 21:23:37.575332 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.575133 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a597eced-76f6-49e8-8176-2e9332188be3-metrics-client-ca\") pod \"thanos-querier-76bbc756f6-s5wtw\" (UID: \"a597eced-76f6-49e8-8176-2e9332188be3\") " pod="openshift-monitoring/thanos-querier-76bbc756f6-s5wtw" Oct 30 21:23:37.575332 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.575139 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-459jm\" (UniqueName: \"kubernetes.io/projected/166e5f35-0987-4eef-917a-a4772acba82f-kube-api-access-459jm\") pod \"kube-state-metrics-5dc8d7d89b-kqxdq\" (UID: \"166e5f35-0987-4eef-917a-a4772acba82f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-kqxdq" Oct 30 21:23:37.575332 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.575167 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-etc-sysctl-conf\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:23:37.575332 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.575216 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-sys\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:23:37.575332 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.575242 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8ea6188f-9e30-4ab0-b5b3-c99647377e00-etc-tuned\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:23:37.575332 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.575286 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/166e5f35-0987-4eef-917a-a4772acba82f-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-kqxdq\" (UID: \"166e5f35-0987-4eef-917a-a4772acba82f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-kqxdq" Oct 30 21:23:37.575332 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.575271 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-cnibin\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:23:37.575332 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.575316 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-host-run-k8s-cni-cncf-io\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:23:37.575332 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.575320 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8ea6188f-9e30-4ab0-b5b3-c99647377e00-etc-tuned\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:23:37.575332 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.575335 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptklg\" (UniqueName: \"kubernetes.io/projected/d9f7709e-2664-452f-b70b-f4649675781e-kube-api-access-ptklg\") pod \"node-resolver-8rbsm\" (UID: \"d9f7709e-2664-452f-b70b-f4649675781e\") " pod="openshift-dns/node-resolver-8rbsm" Oct 30 21:23:37.576160 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.575372 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a597eced-76f6-49e8-8176-2e9332188be3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-76bbc756f6-s5wtw\" (UID: \"a597eced-76f6-49e8-8176-2e9332188be3\") " pod="openshift-monitoring/thanos-querier-76bbc756f6-s5wtw" Oct 30 21:23:37.576160 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.575404 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/aed61d9f-eb67-44ae-a094-a12814e42559-device-dir\") pod \"aws-ebs-csi-driver-node-lvfbv\" (UID: \"aed61d9f-eb67-44ae-a094-a12814e42559\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvfbv" Oct 30 21:23:37.576160 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.575438 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbpln\" (UniqueName: \"kubernetes.io/projected/a9ca84f8-a6d5-45f4-9137-799e05b12c11-kube-api-access-nbpln\") pod \"prometheus-operator-9cd89b74c-fglqz\" (UID: \"a9ca84f8-a6d5-45f4-9137-799e05b12c11\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fglqz" Oct 30 21:23:37.576160 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.575468 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-host-cni-bin\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:23:37.576160 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.575497 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttr7d\" (UniqueName: \"kubernetes.io/projected/a597eced-76f6-49e8-8176-2e9332188be3-kube-api-access-ttr7d\") pod \"thanos-querier-76bbc756f6-s5wtw\" (UID: \"a597eced-76f6-49e8-8176-2e9332188be3\") " pod="openshift-monitoring/thanos-querier-76bbc756f6-s5wtw" Oct 30 21:23:37.576160 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.575524 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ca9f1917-07a8-4ea6-8b59-4f2f99429dfa-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-jfpf9\" (UID: \"ca9f1917-07a8-4ea6-8b59-4f2f99429dfa\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jfpf9" Oct 30 21:23:37.576160 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.575603 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-run\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:23:37.576160 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.575633 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6c9c626a-52a7-466e-b744-f4bd79112fd9-kubelet-config\") pod \"global-pull-secret-syncer-6nncz\" (UID: \"6c9c626a-52a7-466e-b744-f4bd79112fd9\") " pod="kube-system/global-pull-secret-syncer-6nncz" Oct 30 21:23:37.576160 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.575660 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/aed61d9f-eb67-44ae-a094-a12814e42559-registration-dir\") pod \"aws-ebs-csi-driver-node-lvfbv\" (UID: \"aed61d9f-eb67-44ae-a094-a12814e42559\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvfbv" Oct 30 21:23:37.576160 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.575685 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ea6188f-9e30-4ab0-b5b3-c99647377e00-tmp\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:23:37.576160 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.575685 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/59571898-f15b-4b8f-b043-0e5af369c39e-serviceca\") pod \"node-ca-8vqf9\" (UID: \"59571898-f15b-4b8f-b043-0e5af369c39e\") " pod="openshift-image-registry/node-ca-8vqf9" Oct 30 21:23:37.576160 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.575707 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blx28\" (UniqueName: \"kubernetes.io/projected/3cabcccd-6b1e-4089-a3a6-275558de2bfb-kube-api-access-blx28\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:23:37.576160 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.575744 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-host-run-netns\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:23:37.576160 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.575762 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ea6188f-9e30-4ab0-b5b3-c99647377e00-tmp\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:23:37.617837 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.617763 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 30 21:23:37.618687 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.618663 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 21:23:37.618849 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.618785 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 21:23:37.618921 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.618838 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 21:23:37.619031 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.619012 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 21:23:37.619103 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.619032 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 21:23:37.619103 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.619070 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 21:23:37.619103 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.619091 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 21:23:37.619345 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.619316 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 21:23:37.619482 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.619453 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 21:23:37.619546 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.619481 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 21:23:37.619546 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.619496 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 21:23:37.619622 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.619517 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 21:23:37.620321 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.619455 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 21:23:37.620321 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.620045 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 30 21:23:37.620321 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.620016 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 21:23:37.620321 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.620160 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-9qngh\"" Oct 30 21:23:37.620321 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.620020 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 21:23:37.620583 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.620497 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-d6wv5\"" Oct 30 21:23:37.622476 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.620942 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-k65vr\"" Oct 30 21:23:37.622476 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.620966 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 21:23:37.622476 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.621100 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 21:23:37.622476 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.621135 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-wjtfq\"" Oct 30 21:23:37.622476 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.621418 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 21:23:37.622476 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.621497 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-2sgxh\"" Oct 30 21:23:37.622476 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.621735 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 21:23:37.622476 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.621771 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 21:23:37.622476 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.621918 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 21:23:37.622476 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.621933 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 21:23:37.622476 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.621920 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 21:23:37.622476 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.621921 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9ueslha3sv02m\"" Oct 30 21:23:37.622476 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.622240 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 30 21:23:37.622476 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.622278 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 30 21:23:37.623150 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.622542 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 21:23:37.623150 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.622569 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 21:23:37.623150 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.622660 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 30 21:23:37.623150 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.622767 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 21:23:37.623150 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.622796 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-config-volume\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:37.623150 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.622812 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 21:23:37.623150 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.622818 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 30 21:23:37.623150 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.622845 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 21:23:37.623150 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.622879 5748 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:23:37.623150 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:23:37.622946 5748 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-61.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-61.us-west-1.compute.internal" Oct 30 21:23:37.623150 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.622991 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-b7jogd3ihk5l0\"" Oct 30 21:23:37.623150 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.623046 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 21:23:37.623150 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.623094 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-zr9sv\"" Oct 30 21:23:37.623150 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.623104 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 21:23:37.623705 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.623254 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 21:23:37.623705 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.623342 5748 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:23:37.623705 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:23:37.623384 5748 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-61.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-61.us-west-1.compute.internal" Oct 30 21:23:37.623705 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.623437 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-ds517nk6l0lju\"" Oct 30 21:23:37.623705 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.623580 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 21:23:37.623705 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.623583 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b282542e-5b91-430d-bd5e-03af2e875952-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-7qc5s\" (UID: \"b282542e-5b91-430d-bd5e-03af2e875952\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7qc5s" Oct 30 21:23:37.624050 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.623770 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-d49np\"" Oct 30 21:23:37.624050 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.623818 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 21:23:37.624050 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.623854 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 21:23:37.624050 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.623868 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-mg5d2\"" Oct 30 21:23:37.624241 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.624076 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/42aedd5a-da26-4be4-8ed8-01fb25f3ee5f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vfrjv\" (UID: \"42aedd5a-da26-4be4-8ed8-01fb25f3ee5f\") " pod="openshift-insights/insights-runtime-extractor-vfrjv" Oct 30 21:23:37.624241 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.624131 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-lbktx\"" Oct 30 21:23:37.624241 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.624167 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rlt89\" (UID: \"1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979\") " pod="openshift-monitoring/node-exporter-rlt89" Oct 30 21:23:37.624241 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.624164 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a9ca84f8-a6d5-45f4-9137-799e05b12c11-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-fglqz\" (UID: \"a9ca84f8-a6d5-45f4-9137-799e05b12c11\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fglqz" Oct 30 21:23:37.624425 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.624314 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:37.624425 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.624326 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a597eced-76f6-49e8-8176-2e9332188be3-secret-grpc-tls\") pod \"thanos-querier-76bbc756f6-s5wtw\" (UID: \"a597eced-76f6-49e8-8176-2e9332188be3\") " pod="openshift-monitoring/thanos-querier-76bbc756f6-s5wtw" Oct 30 21:23:37.624425 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.624361 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:37.624425 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.624375 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a597eced-76f6-49e8-8176-2e9332188be3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-76bbc756f6-s5wtw\" (UID: \"a597eced-76f6-49e8-8176-2e9332188be3\") " pod="openshift-monitoring/thanos-querier-76bbc756f6-s5wtw" Oct 30 21:23:37.624425 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.624405 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-web-config\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:37.624628 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.624485 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:37.624678 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.624616 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6c9c626a-52a7-466e-b744-f4bd79112fd9-original-pull-secret\") pod \"global-pull-secret-syncer-6nncz\" (UID: \"6c9c626a-52a7-466e-b744-f4bd79112fd9\") " pod="kube-system/global-pull-secret-syncer-6nncz" Oct 30 21:23:37.624814 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.624794 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a597eced-76f6-49e8-8176-2e9332188be3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-76bbc756f6-s5wtw\" (UID: \"a597eced-76f6-49e8-8176-2e9332188be3\") " pod="openshift-monitoring/thanos-querier-76bbc756f6-s5wtw" Oct 30 21:23:37.624979 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.624962 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a597eced-76f6-49e8-8176-2e9332188be3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-76bbc756f6-s5wtw\" (UID: \"a597eced-76f6-49e8-8176-2e9332188be3\") " pod="openshift-monitoring/thanos-querier-76bbc756f6-s5wtw" Oct 30 21:23:37.625117 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.625101 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a9ca84f8-a6d5-45f4-9137-799e05b12c11-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-fglqz\" (UID: \"a9ca84f8-a6d5-45f4-9137-799e05b12c11\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fglqz" Oct 30 21:23:37.625321 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.625278 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:37.625763 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.625744 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a597eced-76f6-49e8-8176-2e9332188be3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-76bbc756f6-s5wtw\" (UID: \"a597eced-76f6-49e8-8176-2e9332188be3\") " pod="openshift-monitoring/thanos-querier-76bbc756f6-s5wtw" Oct 30 21:23:37.625829 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.625771 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ca9f1917-07a8-4ea6-8b59-4f2f99429dfa-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-jfpf9\" (UID: \"ca9f1917-07a8-4ea6-8b59-4f2f99429dfa\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jfpf9" Oct 30 21:23:37.630150 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.630133 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a597eced-76f6-49e8-8176-2e9332188be3-secret-thanos-querier-tls\") pod \"thanos-querier-76bbc756f6-s5wtw\" (UID: \"a597eced-76f6-49e8-8176-2e9332188be3\") " pod="openshift-monitoring/thanos-querier-76bbc756f6-s5wtw" Oct 30 21:23:37.631685 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.631668 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:37.641065 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.641036 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpccd\" (UniqueName: \"kubernetes.io/projected/74930384-19fd-4c0b-b38f-b2f895f7e6e8-kube-api-access-vpccd\") pod \"ingress-canary-zb9t2\" (UID: \"74930384-19fd-4c0b-b38f-b2f895f7e6e8\") " pod="openshift-ingress-canary/ingress-canary-zb9t2" Oct 30 21:23:37.641550 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.641533 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6mc5\" (UniqueName: \"kubernetes.io/projected/6a9841f4-72b0-43fd-8aa7-01393d9a01d1-kube-api-access-g6mc5\") pod \"multus-additional-cni-plugins-jtrcb\" (UID: \"6a9841f4-72b0-43fd-8aa7-01393d9a01d1\") " pod="openshift-multus/multus-additional-cni-plugins-jtrcb" Oct 30 21:23:37.642481 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.642465 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8gxg\" (UniqueName: \"kubernetes.io/projected/83a93696-9f3b-472d-9fc8-92350746df75-kube-api-access-b8gxg\") pod \"network-metrics-daemon-lz8cq\" (UID: \"83a93696-9f3b-472d-9fc8-92350746df75\") " pod="openshift-multus/network-metrics-daemon-lz8cq" Oct 30 21:23:37.643678 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.643661 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpbmf\" (UniqueName: \"kubernetes.io/projected/2496f073-7bf0-48af-b091-be6b964117ca-kube-api-access-vpbmf\") pod \"network-check-target-wv464\" (UID: \"2496f073-7bf0-48af-b091-be6b964117ca\") " pod="openshift-network-diagnostics/network-check-target-wv464" Oct 30 21:23:37.644181 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.644166 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4v76\" (UniqueName: \"kubernetes.io/projected/8ea6188f-9e30-4ab0-b5b3-c99647377e00-kube-api-access-b4v76\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:23:37.645136 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.645122 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r5f6\" (UniqueName: \"kubernetes.io/projected/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-kube-api-access-6r5f6\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:23:37.647079 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.647063 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blx28\" (UniqueName: \"kubernetes.io/projected/3cabcccd-6b1e-4089-a3a6-275558de2bfb-kube-api-access-blx28\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:23:37.648153 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.648138 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km8jp\" (UniqueName: \"kubernetes.io/projected/c2b5bbd1-4e37-4d49-9e28-bdb3c3da309f-kube-api-access-km8jp\") pod \"dns-default-nrm49\" (UID: \"c2b5bbd1-4e37-4d49-9e28-bdb3c3da309f\") " pod="openshift-dns/dns-default-nrm49" Oct 30 21:23:37.650862 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.650845 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4dgx\" (UniqueName: \"kubernetes.io/projected/aed61d9f-eb67-44ae-a094-a12814e42559-kube-api-access-c4dgx\") pod \"aws-ebs-csi-driver-node-lvfbv\" (UID: \"aed61d9f-eb67-44ae-a094-a12814e42559\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvfbv" Oct 30 21:23:37.654446 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.654421 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq8kj\" (UniqueName: \"kubernetes.io/projected/5c3735e3-95ac-4293-8555-38ee9ead0353-kube-api-access-bq8kj\") pod \"iptables-alerter-mxfq6\" (UID: \"5c3735e3-95ac-4293-8555-38ee9ead0353\") " pod="openshift-network-operator/iptables-alerter-mxfq6" Oct 30 21:23:37.656581 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.656557 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbpln\" (UniqueName: \"kubernetes.io/projected/a9ca84f8-a6d5-45f4-9137-799e05b12c11-kube-api-access-nbpln\") pod \"prometheus-operator-9cd89b74c-fglqz\" (UID: \"a9ca84f8-a6d5-45f4-9137-799e05b12c11\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fglqz" Oct 30 21:23:37.656675 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.656655 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw9tn\" (UniqueName: \"kubernetes.io/projected/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-kube-api-access-sw9tn\") pod \"alertmanager-main-0\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:23:37.656763 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.656741 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptklg\" (UniqueName: \"kubernetes.io/projected/d9f7709e-2664-452f-b70b-f4649675781e-kube-api-access-ptklg\") pod \"node-resolver-8rbsm\" (UID: \"d9f7709e-2664-452f-b70b-f4649675781e\") " pod="openshift-dns/node-resolver-8rbsm" Oct 30 21:23:37.656807 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.656771 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2xzl\" (UniqueName: \"kubernetes.io/projected/2dbd490f-d94c-4f12-9106-a12117deda9d-kube-api-access-d2xzl\") pod \"downloads-6dd789c8b7-rq2vl\" (UID: \"2dbd490f-d94c-4f12-9106-a12117deda9d\") " pod="openshift-console/downloads-6dd789c8b7-rq2vl" Oct 30 21:23:37.656869 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.656852 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw8lt\" (UniqueName: \"kubernetes.io/projected/42aedd5a-da26-4be4-8ed8-01fb25f3ee5f-kube-api-access-kw8lt\") pod \"insights-runtime-extractor-vfrjv\" (UID: \"42aedd5a-da26-4be4-8ed8-01fb25f3ee5f\") " pod="openshift-insights/insights-runtime-extractor-vfrjv" Oct 30 21:23:37.658474 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.658443 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-459jm\" (UniqueName: \"kubernetes.io/projected/166e5f35-0987-4eef-917a-a4772acba82f-kube-api-access-459jm\") pod \"kube-state-metrics-5dc8d7d89b-kqxdq\" (UID: \"166e5f35-0987-4eef-917a-a4772acba82f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-kqxdq" Oct 30 21:23:37.660600 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.660584 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs2w6\" (UniqueName: \"kubernetes.io/projected/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979-kube-api-access-hs2w6\") pod \"node-exporter-rlt89\" (UID: \"1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979\") " pod="openshift-monitoring/node-exporter-rlt89" Oct 30 21:23:37.660870 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.660855 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttr7d\" (UniqueName: \"kubernetes.io/projected/a597eced-76f6-49e8-8176-2e9332188be3-kube-api-access-ttr7d\") pod \"thanos-querier-76bbc756f6-s5wtw\" (UID: \"a597eced-76f6-49e8-8176-2e9332188be3\") " pod="openshift-monitoring/thanos-querier-76bbc756f6-s5wtw" Oct 30 21:23:37.661787 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.661772 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7qhc\" (UniqueName: \"kubernetes.io/projected/59571898-f15b-4b8f-b043-0e5af369c39e-kube-api-access-d7qhc\") pod \"node-ca-8vqf9\" (UID: \"59571898-f15b-4b8f-b043-0e5af369c39e\") " pod="openshift-image-registry/node-ca-8vqf9" Oct 30 21:23:37.664355 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.664334 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bghfr\" (UniqueName: \"kubernetes.io/projected/ca9f1917-07a8-4ea6-8b59-4f2f99429dfa-kube-api-access-bghfr\") pod \"openshift-state-metrics-7bd7489957-jfpf9\" (UID: \"ca9f1917-07a8-4ea6-8b59-4f2f99429dfa\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jfpf9" Oct 30 21:23:37.676222 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.676195 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-var-lib-kubelet\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:23:37.676323 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.676226 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-multus-cni-dir\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:23:37.676323 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.676242 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-host-var-lib-cni-bin\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:23:37.676323 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.676261 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:37.676323 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.676278 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/4372da07-b518-42df-bbd7-f148396bba6b-audit-log\") pod \"metrics-server-5dcb975b97-crpzf\" (UID: \"4372da07-b518-42df-bbd7-f148396bba6b\") " pod="openshift-monitoring/metrics-server-5dcb975b97-crpzf" Oct 30 21:23:37.676323 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.676300 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c3735e3-95ac-4293-8555-38ee9ead0353-host-slash\") pod \"iptables-alerter-mxfq6\" (UID: \"5c3735e3-95ac-4293-8555-38ee9ead0353\") " pod="openshift-network-operator/iptables-alerter-mxfq6" Oct 30 21:23:37.676323 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.676314 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-multus-cni-dir\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:23:37.676323 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.676316 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-var-lib-kubelet\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:23:37.676584 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.676326 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/aed61d9f-eb67-44ae-a094-a12814e42559-etc-selinux\") pod \"aws-ebs-csi-driver-node-lvfbv\" (UID: \"aed61d9f-eb67-44ae-a094-a12814e42559\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvfbv" Oct 30 21:23:37.676584 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.676331 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-host-var-lib-cni-bin\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:23:37.676584 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.676392 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/aed61d9f-eb67-44ae-a094-a12814e42559-etc-selinux\") pod \"aws-ebs-csi-driver-node-lvfbv\" (UID: \"aed61d9f-eb67-44ae-a094-a12814e42559\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvfbv" Oct 30 21:23:37.676584 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.676405 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:37.676584 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.676423 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c3735e3-95ac-4293-8555-38ee9ead0353-host-slash\") pod \"iptables-alerter-mxfq6\" (UID: \"5c3735e3-95ac-4293-8555-38ee9ead0353\") " pod="openshift-network-operator/iptables-alerter-mxfq6" Oct 30 21:23:37.676584 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.676408 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/4372da07-b518-42df-bbd7-f148396bba6b-audit-log\") pod \"metrics-server-5dcb975b97-crpzf\" (UID: \"4372da07-b518-42df-bbd7-f148396bba6b\") " pod="openshift-monitoring/metrics-server-5dcb975b97-crpzf" Oct 30 21:23:37.676584 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.676466 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-run-openvswitch\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:23:37.676584 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.676490 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-run-openvswitch\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:23:37.676584 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.676504 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:37.676584 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.676534 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/aed61d9f-eb67-44ae-a094-a12814e42559-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lvfbv\" (UID: \"aed61d9f-eb67-44ae-a094-a12814e42559\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvfbv" Oct 30 21:23:37.676584 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.676574 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/aed61d9f-eb67-44ae-a094-a12814e42559-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lvfbv\" (UID: \"aed61d9f-eb67-44ae-a094-a12814e42559\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvfbv" Oct 30 21:23:37.677022 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.676640 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-host\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:23:37.677022 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.676662 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4372da07-b518-42df-bbd7-f148396bba6b-client-ca-bundle\") pod \"metrics-server-5dcb975b97-crpzf\" (UID: \"4372da07-b518-42df-bbd7-f148396bba6b\") " pod="openshift-monitoring/metrics-server-5dcb975b97-crpzf" Oct 30 21:23:37.677022 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.676677 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6a9841f4-72b0-43fd-8aa7-01393d9a01d1-cnibin\") pod \"multus-additional-cni-plugins-jtrcb\" (UID: \"6a9841f4-72b0-43fd-8aa7-01393d9a01d1\") " pod="openshift-multus/multus-additional-cni-plugins-jtrcb" Oct 30 21:23:37.677022 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.676692 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-etc-systemd\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:23:37.677022 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.676705 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-system-cni-dir\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:23:37.677022 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.676720 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-run-systemd\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:23:37.677022 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.676737 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a9841f4-72b0-43fd-8aa7-01393d9a01d1-system-cni-dir\") pod \"multus-additional-cni-plugins-jtrcb\" (UID: \"6a9841f4-72b0-43fd-8aa7-01393d9a01d1\") " pod="openshift-multus/multus-additional-cni-plugins-jtrcb" Oct 30 21:23:37.677022 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.676754 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6a9841f4-72b0-43fd-8aa7-01393d9a01d1-cnibin\") pod \"multus-additional-cni-plugins-jtrcb\" (UID: \"6a9841f4-72b0-43fd-8aa7-01393d9a01d1\") " pod="openshift-multus/multus-additional-cni-plugins-jtrcb" Oct 30 21:23:37.677022 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.676761 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-host\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:23:37.677022 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.676761 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-multus-conf-dir\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:23:37.677022 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.676783 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-system-cni-dir\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:23:37.677022 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.676797 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-run-systemd\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:23:37.677022 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.676801 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-host-slash\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:23:37.677022 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.676826 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-node-log\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:23:37.677022 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.676835 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-host-slash\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:23:37.677022 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.676865 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-host-run-ovn-kubernetes\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:23:37.677022 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.676871 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a9841f4-72b0-43fd-8aa7-01393d9a01d1-system-cni-dir\") pod \"multus-additional-cni-plugins-jtrcb\" (UID: \"6a9841f4-72b0-43fd-8aa7-01393d9a01d1\") " pod="openshift-multus/multus-additional-cni-plugins-jtrcb" Oct 30 21:23:37.677022 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.676872 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-etc-systemd\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:23:37.677022 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.676909 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-host-run-ovn-kubernetes\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:23:37.677022 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.676916 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-web-config\") pod \"prometheus-k8s-0\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:37.677022 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.676924 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4372da07-b518-42df-bbd7-f148396bba6b-client-ca-bundle\") pod \"metrics-server-5dcb975b97-crpzf\" (UID: \"4372da07-b518-42df-bbd7-f148396bba6b\") " pod="openshift-monitoring/metrics-server-5dcb975b97-crpzf" Oct 30 21:23:37.677022 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.676921 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-node-log\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:23:37.677022 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.676923 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-multus-conf-dir\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:23:37.677022 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.676949 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-host-run-multus-certs\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:23:37.677022 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.676991 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:37.677022 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.677003 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-host-run-multus-certs\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:23:37.677915 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.677052 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d9f7709e-2664-452f-b70b-f4649675781e-hosts-file\") pod \"node-resolver-8rbsm\" (UID: \"d9f7709e-2664-452f-b70b-f4649675781e\") " pod="openshift-dns/node-resolver-8rbsm" Oct 30 21:23:37.677915 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.677070 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979-sys\") pod \"node-exporter-rlt89\" (UID: \"1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979\") " pod="openshift-monitoring/node-exporter-rlt89" Oct 30 21:23:37.677915 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.677101 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d9f7709e-2664-452f-b70b-f4649675781e-hosts-file\") pod \"node-resolver-8rbsm\" (UID: \"d9f7709e-2664-452f-b70b-f4649675781e\") " pod="openshift-dns/node-resolver-8rbsm" Oct 30 21:23:37.677915 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.677103 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979-sys\") pod \"node-exporter-rlt89\" (UID: \"1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979\") " pod="openshift-monitoring/node-exporter-rlt89" Oct 30 21:23:37.677915 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.677126 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbkld\" (UniqueName: \"kubernetes.io/projected/f0d8d720-b6f1-4600-970f-8191318c3e3e-kube-api-access-lbkld\") pod \"monitoring-plugin-77f4f76594-2r74x\" (UID: \"f0d8d720-b6f1-4600-970f-8191318c3e3e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2r74x" Oct 30 21:23:37.677915 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.677143 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d8f2a8e7-53db-4867-bd38-a8d45b15e40d-secret-telemeter-client\") pod \"telemeter-client-7c58f6d48-ls9th\" (UID: \"d8f2a8e7-53db-4867-bd38-a8d45b15e40d\") " pod="openshift-monitoring/telemeter-client-7c58f6d48-ls9th" Oct 30 21:23:37.677915 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.677158 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-console-config\") pod \"console-7fbd54bb44-d7zl5\" (UID: \"30b586ee-8d8a-4e95-b9a3-64f3b09c77e6\") " pod="openshift-console/console-7fbd54bb44-d7zl5" Oct 30 21:23:37.677915 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.677172 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-service-ca\") pod \"console-7fbd54bb44-d7zl5\" (UID: \"30b586ee-8d8a-4e95-b9a3-64f3b09c77e6\") " pod="openshift-console/console-7fbd54bb44-d7zl5" Oct 30 21:23:37.677915 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.677185 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-log-socket\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:23:37.677915 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.677209 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-log-socket\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:23:37.677915 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.677302 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nk9v\" (UniqueName: \"kubernetes.io/projected/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-kube-api-access-4nk9v\") pod \"console-7fbd54bb44-d7zl5\" (UID: \"30b586ee-8d8a-4e95-b9a3-64f3b09c77e6\") " pod="openshift-console/console-7fbd54bb44-d7zl5" Oct 30 21:23:37.677915 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.677334 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:37.677915 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.677362 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d8f2a8e7-53db-4867-bd38-a8d45b15e40d-secret-telemeter-client\") pod \"telemeter-client-7c58f6d48-ls9th\" (UID: \"d8f2a8e7-53db-4867-bd38-a8d45b15e40d\") " pod="openshift-monitoring/telemeter-client-7c58f6d48-ls9th" Oct 30 21:23:37.677915 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.677369 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:37.677915 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.677393 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-service-ca\") pod \"console-7fbd54bb44-d7zl5\" (UID: \"30b586ee-8d8a-4e95-b9a3-64f3b09c77e6\") " pod="openshift-console/console-7fbd54bb44-d7zl5" Oct 30 21:23:37.677915 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.677394 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-console-config\") pod \"console-7fbd54bb44-d7zl5\" (UID: \"30b586ee-8d8a-4e95-b9a3-64f3b09c77e6\") " pod="openshift-console/console-7fbd54bb44-d7zl5" Oct 30 21:23:37.677915 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.677423 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:37.677915 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.677473 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzjnx\" (UniqueName: \"kubernetes.io/projected/d8f2a8e7-53db-4867-bd38-a8d45b15e40d-kube-api-access-dzjnx\") pod \"telemeter-client-7c58f6d48-ls9th\" (UID: \"d8f2a8e7-53db-4867-bd38-a8d45b15e40d\") " pod="openshift-monitoring/telemeter-client-7c58f6d48-ls9th" Oct 30 21:23:37.677915 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.677492 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-console-oauth-config\") pod \"console-7fbd54bb44-d7zl5\" (UID: \"30b586ee-8d8a-4e95-b9a3-64f3b09c77e6\") " pod="openshift-console/console-7fbd54bb44-d7zl5" Oct 30 21:23:37.677915 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.677509 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:37.677915 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.677524 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:37.677915 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.677543 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:37.677915 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.677575 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:37.677915 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.677624 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:37.677915 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.677660 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9vqk\" (UniqueName: \"kubernetes.io/projected/4372da07-b518-42df-bbd7-f148396bba6b-kube-api-access-x9vqk\") pod \"metrics-server-5dcb975b97-crpzf\" (UID: \"4372da07-b518-42df-bbd7-f148396bba6b\") " pod="openshift-monitoring/metrics-server-5dcb975b97-crpzf" Oct 30 21:23:37.677915 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.677666 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-console-oauth-config\") pod \"console-7fbd54bb44-d7zl5\" (UID: \"30b586ee-8d8a-4e95-b9a3-64f3b09c77e6\") " pod="openshift-console/console-7fbd54bb44-d7zl5" Oct 30 21:23:37.677915 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.677698 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979-root\") pod \"node-exporter-rlt89\" (UID: \"1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979\") " pod="openshift-monitoring/node-exporter-rlt89" Oct 30 21:23:37.677915 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.677725 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-tmp\") pod \"console-7fbd54bb44-d7zl5\" (UID: \"30b586ee-8d8a-4e95-b9a3-64f3b09c77e6\") " pod="openshift-console/console-7fbd54bb44-d7zl5" Oct 30 21:23:37.677915 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.677752 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/4372da07-b518-42df-bbd7-f148396bba6b-secret-metrics-server-client-certs\") pod \"metrics-server-5dcb975b97-crpzf\" (UID: \"4372da07-b518-42df-bbd7-f148396bba6b\") " pod="openshift-monitoring/metrics-server-5dcb975b97-crpzf" Oct 30 21:23:37.677915 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.677784 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979-root\") pod \"node-exporter-rlt89\" (UID: \"1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979\") " pod="openshift-monitoring/node-exporter-rlt89" Oct 30 21:23:37.677915 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.677797 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-etc-kubernetes\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:23:37.677915 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.677821 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-tmp\") pod \"console-7fbd54bb44-d7zl5\" (UID: \"30b586ee-8d8a-4e95-b9a3-64f3b09c77e6\") " pod="openshift-console/console-7fbd54bb44-d7zl5" Oct 30 21:23:37.677915 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.677825 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-etc-kubernetes\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:23:37.677915 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.677851 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-etc-kubernetes\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:23:37.677915 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.677866 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-console-serving-cert\") pod \"console-7fbd54bb44-d7zl5\" (UID: \"30b586ee-8d8a-4e95-b9a3-64f3b09c77e6\") " pod="openshift-console/console-7fbd54bb44-d7zl5" Oct 30 21:23:37.677915 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.677922 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-etc-kubernetes\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:23:37.677915 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.677912 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/4372da07-b518-42df-bbd7-f148396bba6b-secret-metrics-server-tls\") pod \"metrics-server-5dcb975b97-crpzf\" (UID: \"4372da07-b518-42df-bbd7-f148396bba6b\") " pod="openshift-monitoring/metrics-server-5dcb975b97-crpzf" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.677968 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-host-kubelet\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.677974 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/4372da07-b518-42df-bbd7-f148396bba6b-secret-metrics-server-client-certs\") pod \"metrics-server-5dcb975b97-crpzf\" (UID: \"4372da07-b518-42df-bbd7-f148396bba6b\") " pod="openshift-monitoring/metrics-server-5dcb975b97-crpzf" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.677995 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.678010 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-host-kubelet\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.678025 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-etc-sysctl-d\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.678049 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6c9c626a-52a7-466e-b744-f4bd79112fd9-dbus\") pod \"global-pull-secret-syncer-6nncz\" (UID: \"6c9c626a-52a7-466e-b744-f4bd79112fd9\") " pod="kube-system/global-pull-secret-syncer-6nncz" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.678073 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-var-lib-openvswitch\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.678103 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d8f2a8e7-53db-4867-bd38-a8d45b15e40d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7c58f6d48-ls9th\" (UID: \"d8f2a8e7-53db-4867-bd38-a8d45b15e40d\") " pod="openshift-monitoring/telemeter-client-7c58f6d48-ls9th" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.678114 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-etc-sysctl-d\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.678124 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6c9c626a-52a7-466e-b744-f4bd79112fd9-dbus\") pod \"global-pull-secret-syncer-6nncz\" (UID: \"6c9c626a-52a7-466e-b744-f4bd79112fd9\") " pod="kube-system/global-pull-secret-syncer-6nncz" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.678104 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-console-serving-cert\") pod \"console-7fbd54bb44-d7zl5\" (UID: \"30b586ee-8d8a-4e95-b9a3-64f3b09c77e6\") " pod="openshift-console/console-7fbd54bb44-d7zl5" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.678142 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-var-lib-openvswitch\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.678134 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8f2a8e7-53db-4867-bd38-a8d45b15e40d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7c58f6d48-ls9th\" (UID: \"d8f2a8e7-53db-4867-bd38-a8d45b15e40d\") " pod="openshift-monitoring/telemeter-client-7c58f6d48-ls9th" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.678197 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-run-ovn\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.678144 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/4372da07-b518-42df-bbd7-f148396bba6b-secret-metrics-server-tls\") pod \"metrics-server-5dcb975b97-crpzf\" (UID: \"4372da07-b518-42df-bbd7-f148396bba6b\") " pod="openshift-monitoring/metrics-server-5dcb975b97-crpzf" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.678222 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d8f2a8e7-53db-4867-bd38-a8d45b15e40d-federate-client-tls\") pod \"telemeter-client-7c58f6d48-ls9th\" (UID: \"d8f2a8e7-53db-4867-bd38-a8d45b15e40d\") " pod="openshift-monitoring/telemeter-client-7c58f6d48-ls9th" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.678251 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d8f2a8e7-53db-4867-bd38-a8d45b15e40d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7c58f6d48-ls9th\" (UID: \"d8f2a8e7-53db-4867-bd38-a8d45b15e40d\") " pod="openshift-monitoring/telemeter-client-7c58f6d48-ls9th" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.678268 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979-node-exporter-wtmp\") pod \"node-exporter-rlt89\" (UID: \"1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979\") " pod="openshift-monitoring/node-exporter-rlt89" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.678286 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-lib-modules\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.678299 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979-node-exporter-wtmp\") pod \"node-exporter-rlt89\" (UID: \"1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979\") " pod="openshift-monitoring/node-exporter-rlt89" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.678267 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-run-ovn\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.678326 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-hostroot\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.678301 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-hostroot\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.678378 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4372da07-b518-42df-bbd7-f148396bba6b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5dcb975b97-crpzf\" (UID: \"4372da07-b518-42df-bbd7-f148396bba6b\") " pod="openshift-monitoring/metrics-server-5dcb975b97-crpzf" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.678410 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8f2a8e7-53db-4867-bd38-a8d45b15e40d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7c58f6d48-ls9th\" (UID: \"d8f2a8e7-53db-4867-bd38-a8d45b15e40d\") " pod="openshift-monitoring/telemeter-client-7c58f6d48-ls9th" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.678417 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d8f2a8e7-53db-4867-bd38-a8d45b15e40d-federate-client-tls\") pod \"telemeter-client-7c58f6d48-ls9th\" (UID: \"d8f2a8e7-53db-4867-bd38-a8d45b15e40d\") " pod="openshift-monitoring/telemeter-client-7c58f6d48-ls9th" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.678414 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.678383 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-lib-modules\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.678468 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jngrf\" (UniqueName: \"kubernetes.io/projected/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-kube-api-access-jngrf\") pod \"prometheus-k8s-0\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.678489 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/59571898-f15b-4b8f-b043-0e5af369c39e-host\") pod \"node-ca-8vqf9\" (UID: \"59571898-f15b-4b8f-b043-0e5af369c39e\") " pod="openshift-image-registry/node-ca-8vqf9" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.678524 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.678528 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4372da07-b518-42df-bbd7-f148396bba6b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5dcb975b97-crpzf\" (UID: \"4372da07-b518-42df-bbd7-f148396bba6b\") " pod="openshift-monitoring/metrics-server-5dcb975b97-crpzf" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.678578 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/59571898-f15b-4b8f-b043-0e5af369c39e-host\") pod \"node-ca-8vqf9\" (UID: \"59571898-f15b-4b8f-b043-0e5af369c39e\") " pod="openshift-image-registry/node-ca-8vqf9" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.678580 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.678617 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.678643 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-host-run-netns\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.678669 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-oauth-serving-cert\") pod \"console-7fbd54bb44-d7zl5\" (UID: \"30b586ee-8d8a-4e95-b9a3-64f3b09c77e6\") " pod="openshift-console/console-7fbd54bb44-d7zl5" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.678711 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-host-run-netns\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.678720 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/4372da07-b518-42df-bbd7-f148396bba6b-metrics-server-audit-profiles\") pod \"metrics-server-5dcb975b97-crpzf\" (UID: \"4372da07-b518-42df-bbd7-f148396bba6b\") " pod="openshift-monitoring/metrics-server-5dcb975b97-crpzf" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.678749 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6a9841f4-72b0-43fd-8aa7-01393d9a01d1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jtrcb\" (UID: \"6a9841f4-72b0-43fd-8aa7-01393d9a01d1\") " pod="openshift-multus/multus-additional-cni-plugins-jtrcb" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.678776 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/aed61d9f-eb67-44ae-a094-a12814e42559-sys-fs\") pod \"aws-ebs-csi-driver-node-lvfbv\" (UID: \"aed61d9f-eb67-44ae-a094-a12814e42559\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvfbv" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.678801 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-host-var-lib-cni-multus\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.678827 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d8f2a8e7-53db-4867-bd38-a8d45b15e40d-telemeter-client-tls\") pod \"telemeter-client-7c58f6d48-ls9th\" (UID: \"d8f2a8e7-53db-4867-bd38-a8d45b15e40d\") " pod="openshift-monitoring/telemeter-client-7c58f6d48-ls9th" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.678854 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-etc-sysctl-conf\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.678864 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/aed61d9f-eb67-44ae-a094-a12814e42559-sys-fs\") pod \"aws-ebs-csi-driver-node-lvfbv\" (UID: \"aed61d9f-eb67-44ae-a094-a12814e42559\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvfbv" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.678874 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6a9841f4-72b0-43fd-8aa7-01393d9a01d1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jtrcb\" (UID: \"6a9841f4-72b0-43fd-8aa7-01393d9a01d1\") " pod="openshift-multus/multus-additional-cni-plugins-jtrcb" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.678876 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-sys\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.678874 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-host-var-lib-cni-multus\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.678939 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-oauth-serving-cert\") pod \"console-7fbd54bb44-d7zl5\" (UID: \"30b586ee-8d8a-4e95-b9a3-64f3b09c77e6\") " pod="openshift-console/console-7fbd54bb44-d7zl5" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.678948 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-sys\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.678974 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-cnibin\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.678990 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-host-run-k8s-cni-cncf-io\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.679006 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-config\") pod \"prometheus-k8s-0\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.679013 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-etc-sysctl-conf\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.679028 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/aed61d9f-eb67-44ae-a094-a12814e42559-device-dir\") pod \"aws-ebs-csi-driver-node-lvfbv\" (UID: \"aed61d9f-eb67-44ae-a094-a12814e42559\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvfbv" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.679041 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-cnibin\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.679043 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/4372da07-b518-42df-bbd7-f148396bba6b-metrics-server-audit-profiles\") pod \"metrics-server-5dcb975b97-crpzf\" (UID: \"4372da07-b518-42df-bbd7-f148396bba6b\") " pod="openshift-monitoring/metrics-server-5dcb975b97-crpzf" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.679047 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-host-cni-bin\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.679065 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-host-run-k8s-cni-cncf-io\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.679049 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d8f2a8e7-53db-4867-bd38-a8d45b15e40d-telemeter-client-tls\") pod \"telemeter-client-7c58f6d48-ls9th\" (UID: \"d8f2a8e7-53db-4867-bd38-a8d45b15e40d\") " pod="openshift-monitoring/telemeter-client-7c58f6d48-ls9th" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.679108 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-host-cni-bin\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.679108 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.679134 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/aed61d9f-eb67-44ae-a094-a12814e42559-device-dir\") pod \"aws-ebs-csi-driver-node-lvfbv\" (UID: \"aed61d9f-eb67-44ae-a094-a12814e42559\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvfbv" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.679174 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-run\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.679195 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6c9c626a-52a7-466e-b744-f4bd79112fd9-kubelet-config\") pod \"global-pull-secret-syncer-6nncz\" (UID: \"6c9c626a-52a7-466e-b744-f4bd79112fd9\") " pod="kube-system/global-pull-secret-syncer-6nncz" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.679230 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-run\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.679231 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/aed61d9f-eb67-44ae-a094-a12814e42559-registration-dir\") pod \"aws-ebs-csi-driver-node-lvfbv\" (UID: \"aed61d9f-eb67-44ae-a094-a12814e42559\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvfbv" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.679264 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/aed61d9f-eb67-44ae-a094-a12814e42559-registration-dir\") pod \"aws-ebs-csi-driver-node-lvfbv\" (UID: \"aed61d9f-eb67-44ae-a094-a12814e42559\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvfbv" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.679278 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6c9c626a-52a7-466e-b744-f4bd79112fd9-kubelet-config\") pod \"global-pull-secret-syncer-6nncz\" (UID: \"6c9c626a-52a7-466e-b744-f4bd79112fd9\") " pod="kube-system/global-pull-secret-syncer-6nncz" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.679303 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-host-run-netns\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.679336 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-host-cni-netd\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.679366 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f0d8d720-b6f1-4600-970f-8191318c3e3e-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-2r74x\" (UID: \"f0d8d720-b6f1-4600-970f-8191318c3e3e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2r74x" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.679375 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-host-run-netns\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.679394 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8f2a8e7-53db-4867-bd38-a8d45b15e40d-serving-certs-ca-bundle\") pod \"telemeter-client-7c58f6d48-ls9th\" (UID: \"d8f2a8e7-53db-4867-bd38-a8d45b15e40d\") " pod="openshift-monitoring/telemeter-client-7c58f6d48-ls9th" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.679404 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-host-cni-netd\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:23:37.679460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.679421 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8f2a8e7-53db-4867-bd38-a8d45b15e40d-metrics-client-ca\") pod \"telemeter-client-7c58f6d48-ls9th\" (UID: \"d8f2a8e7-53db-4867-bd38-a8d45b15e40d\") " pod="openshift-monitoring/telemeter-client-7c58f6d48-ls9th" Oct 30 21:23:37.681616 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.679527 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/aed61d9f-eb67-44ae-a094-a12814e42559-socket-dir\") pod \"aws-ebs-csi-driver-node-lvfbv\" (UID: \"aed61d9f-eb67-44ae-a094-a12814e42559\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvfbv" Oct 30 21:23:37.681616 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.679536 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8f2a8e7-53db-4867-bd38-a8d45b15e40d-serving-certs-ca-bundle\") pod \"telemeter-client-7c58f6d48-ls9th\" (UID: \"d8f2a8e7-53db-4867-bd38-a8d45b15e40d\") " pod="openshift-monitoring/telemeter-client-7c58f6d48-ls9th" Oct 30 21:23:37.681616 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.679552 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/42aedd5a-da26-4be4-8ed8-01fb25f3ee5f-crio-socket\") pod \"insights-runtime-extractor-vfrjv\" (UID: \"42aedd5a-da26-4be4-8ed8-01fb25f3ee5f\") " pod="openshift-insights/insights-runtime-extractor-vfrjv" Oct 30 21:23:37.681616 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.679574 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/aed61d9f-eb67-44ae-a094-a12814e42559-socket-dir\") pod \"aws-ebs-csi-driver-node-lvfbv\" (UID: \"aed61d9f-eb67-44ae-a094-a12814e42559\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvfbv" Oct 30 21:23:37.681616 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.679575 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-etc-sysconfig\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:23:37.681616 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.679591 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f0d8d720-b6f1-4600-970f-8191318c3e3e-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-2r74x\" (UID: \"f0d8d720-b6f1-4600-970f-8191318c3e3e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2r74x" Oct 30 21:23:37.681616 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.679600 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-systemd-units\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:23:37.681616 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.679609 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/42aedd5a-da26-4be4-8ed8-01fb25f3ee5f-crio-socket\") pod \"insights-runtime-extractor-vfrjv\" (UID: \"42aedd5a-da26-4be4-8ed8-01fb25f3ee5f\") " pod="openshift-insights/insights-runtime-extractor-vfrjv" Oct 30 21:23:37.681616 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.679624 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-etc-openvswitch\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:23:37.681616 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.679635 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-systemd-units\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:23:37.681616 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.679651 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-etc-sysconfig\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:23:37.681616 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.679651 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-multus-socket-dir-parent\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:23:37.681616 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.679672 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8f2a8e7-53db-4867-bd38-a8d45b15e40d-metrics-client-ca\") pod \"telemeter-client-7c58f6d48-ls9th\" (UID: \"d8f2a8e7-53db-4867-bd38-a8d45b15e40d\") " pod="openshift-monitoring/telemeter-client-7c58f6d48-ls9th" Oct 30 21:23:37.681616 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.679689 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-host-var-lib-kubelet\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:23:37.681616 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.679697 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-multus-socket-dir-parent\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:23:37.681616 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.679666 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3df036ff-1c00-4ce2-9e86-8acbeedf6c82-etc-openvswitch\") pod \"ovnkube-node-j4v8r\" (UID: \"3df036ff-1c00-4ce2-9e86-8acbeedf6c82\") " pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:23:37.681616 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.679712 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-config-out\") pod \"prometheus-k8s-0\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:37.681616 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.679722 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3cabcccd-6b1e-4089-a3a6-275558de2bfb-host-var-lib-kubelet\") pod \"multus-n2sfk\" (UID: \"3cabcccd-6b1e-4089-a3a6-275558de2bfb\") " pod="openshift-multus/multus-n2sfk" Oct 30 21:23:37.681616 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.679748 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-etc-modprobe-d\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:23:37.681616 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.679847 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8ea6188f-9e30-4ab0-b5b3-c99647377e00-etc-modprobe-d\") pod \"tuned-dpr6c\" (UID: \"8ea6188f-9e30-4ab0-b5b3-c99647377e00\") " pod="openshift-cluster-node-tuning-operator/tuned-dpr6c" Oct 30 21:23:37.744076 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.744041 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9vqk\" (UniqueName: \"kubernetes.io/projected/4372da07-b518-42df-bbd7-f148396bba6b-kube-api-access-x9vqk\") pod \"metrics-server-5dcb975b97-crpzf\" (UID: \"4372da07-b518-42df-bbd7-f148396bba6b\") " pod="openshift-monitoring/metrics-server-5dcb975b97-crpzf" Oct 30 21:23:37.744278 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.744213 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbkld\" (UniqueName: \"kubernetes.io/projected/f0d8d720-b6f1-4600-970f-8191318c3e3e-kube-api-access-lbkld\") pod \"monitoring-plugin-77f4f76594-2r74x\" (UID: \"f0d8d720-b6f1-4600-970f-8191318c3e3e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2r74x" Oct 30 21:23:37.744278 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.744215 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzjnx\" (UniqueName: \"kubernetes.io/projected/d8f2a8e7-53db-4867-bd38-a8d45b15e40d-kube-api-access-dzjnx\") pod \"telemeter-client-7c58f6d48-ls9th\" (UID: \"d8f2a8e7-53db-4867-bd38-a8d45b15e40d\") " pod="openshift-monitoring/telemeter-client-7c58f6d48-ls9th" Oct 30 21:23:37.744587 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.744566 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nk9v\" (UniqueName: \"kubernetes.io/projected/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-kube-api-access-4nk9v\") pod \"console-7fbd54bb44-d7zl5\" (UID: \"30b586ee-8d8a-4e95-b9a3-64f3b09c77e6\") " pod="openshift-console/console-7fbd54bb44-d7zl5" Oct 30 21:23:37.770687 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.770667 5748 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:23:37.780169 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.780147 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-web-config\") pod \"prometheus-k8s-0\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:37.780248 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.780178 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:37.780248 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.780205 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:37.780248 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.780233 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:37.780386 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.780255 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:37.780386 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.780279 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:37.780386 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.780305 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:37.780386 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.780331 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:37.780386 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.780356 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:37.780386 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.780379 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:37.780643 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.780413 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:37.780643 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.780453 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:37.780643 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.780476 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jngrf\" (UniqueName: \"kubernetes.io/projected/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-kube-api-access-jngrf\") pod \"prometheus-k8s-0\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:37.780643 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.780502 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:37.780643 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.780547 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-config\") pod \"prometheus-k8s-0\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:37.780643 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.780565 5748 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 21:23:37.780643 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.780574 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:37.780643 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.780618 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-config-out\") pod \"prometheus-k8s-0\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:37.781040 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.780651 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:37.781040 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.780679 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:37.781040 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.780737 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:37.781040 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.780993 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:37.781242 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.781183 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:37.781343 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.781294 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:37.781532 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.781508 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:37.782035 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.782011 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:37.782891 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.782837 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:37.783566 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.783545 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:37.783657 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.783589 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:37.783657 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.783633 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:37.783805 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.783780 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:37.784001 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.783981 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:37.785252 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.785223 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:37.785488 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.785462 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:37.785621 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.785600 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-config\") pod \"prometheus-k8s-0\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:37.785937 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.785891 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-web-config\") pod \"prometheus-k8s-0\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:37.786168 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.786148 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:37.786229 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.786201 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:37.786282 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.786267 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-config-out\") pod \"prometheus-k8s-0\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:37.786607 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.786588 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:37.844726 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:37.844688 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jngrf\" (UniqueName: \"kubernetes.io/projected/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-kube-api-access-jngrf\") pod \"prometheus-k8s-0\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:38.107291 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:38.107242 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:38.607657 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:38.607626 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:23:38.663635 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:23:38.663594 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbce4cfcf_5926_433e_a0dd_238dfbb6ef1c.slice/crio-5a5ce108388966ac477aac72c60f47a51638b2b20925a37abb6f5235903187fc WatchSource:0}: Error finding container 5a5ce108388966ac477aac72c60f47a51638b2b20925a37abb6f5235903187fc: Status 404 returned error can't find the container with id 5a5ce108388966ac477aac72c60f47a51638b2b20925a37abb6f5235903187fc Oct 30 21:23:39.100951 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:39.100240 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7qc5s" Oct 30 21:23:39.151724 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:39.151139 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-nrm49" Oct 30 21:23:39.509449 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:39.509414 5748 generic.go:356] "Generic (PLEG): container finished" podID="bce4cfcf-5926-433e-a0dd-238dfbb6ef1c" containerID="afe6661de93bfc5e26c209186b3db8d4b82dbe82485e926e05bf19ad430201de" exitCode=0 Oct 30 21:23:39.509687 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:39.509523 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c","Type":"ContainerDied","Data":"afe6661de93bfc5e26c209186b3db8d4b82dbe82485e926e05bf19ad430201de"} Oct 30 21:23:39.509764 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:39.509665 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c","Type":"ContainerStarted","Data":"5a5ce108388966ac477aac72c60f47a51638b2b20925a37abb6f5235903187fc"} Oct 30 21:23:39.516740 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:39.516688 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1","Type":"ContainerStarted","Data":"1ab7380b3c86066a0ea469c93099103fcbe350a2670812a741bd38f032e50bd6"} Oct 30 21:23:39.516862 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:39.516829 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1","Type":"ContainerStarted","Data":"1839c3a83541f7d614e579209dc5e9561086789aa379aa32c22e64d32f1a3b57"} Oct 30 21:23:39.516949 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:39.516887 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1","Type":"ContainerStarted","Data":"3d284468d6c415172978473132a6e83ab9c2ecb67d9f97053b01fa3b62b871e0"} Oct 30 21:23:39.517026 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:39.516960 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1","Type":"ContainerStarted","Data":"a0785c5b20f085f4118b42cdad2c706fe576eb55fe8dfc18165491e49e533545"} Oct 30 21:23:39.517026 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:39.517016 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1","Type":"ContainerStarted","Data":"da7a54e5c2563fc81cc86e63f88843464d1bd9606e0b7d6ffd0e8836db6677ef"} Oct 30 21:23:39.517115 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:39.517070 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1","Type":"ContainerStarted","Data":"7297c874e6923540bf55507098ad23fe6dfda950450aa8d7b337c2721b4171d7"} Oct 30 21:23:39.522945 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:39.522914 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76bbc756f6-s5wtw" event={"ID":"a597eced-76f6-49e8-8176-2e9332188be3","Type":"ContainerStarted","Data":"fdb797b5d4c5b7ee3fb844676736f5a6b666d087b64188960ab2b8c9ad31dee8"} Oct 30 21:23:39.523066 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:39.522985 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76bbc756f6-s5wtw" event={"ID":"a597eced-76f6-49e8-8176-2e9332188be3","Type":"ContainerStarted","Data":"305373e4cee05a7069d69b7c33f8b345fefa65e1b46fc5e87c63d9f930cd9fa9"} Oct 30 21:23:39.523066 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:39.523024 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76bbc756f6-s5wtw" event={"ID":"a597eced-76f6-49e8-8176-2e9332188be3","Type":"ContainerStarted","Data":"a170677cb5a7ad760372ed06898bf642ea82ec152bd2ccd5de14ff69c80fbdfc"} Oct 30 21:23:39.700934 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:39.700884 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wv464" Oct 30 21:23:39.823488 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:39.823385 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-76bbc756f6-s5wtw" Oct 30 21:23:39.831584 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:39.831550 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-76bbc756f6-s5wtw" Oct 30 21:23:40.441706 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:40.441672 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:23:41.538693 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:41.538285 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7c58f6d48-ls9th" event={"ID":"d8f2a8e7-53db-4867-bd38-a8d45b15e40d","Type":"ContainerStarted","Data":"2e81b075bc5887831a84ead5af0a3985308f718cf469006bae3c1243e7279b5e"} Oct 30 21:23:41.538693 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:41.538369 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7c58f6d48-ls9th" event={"ID":"d8f2a8e7-53db-4867-bd38-a8d45b15e40d","Type":"ContainerStarted","Data":"58f384bdafcea20b4914cfec424292a67162cb38f8782f57b0d45071cca8335c"} Oct 30 21:23:41.538693 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:41.538390 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7c58f6d48-ls9th" event={"ID":"d8f2a8e7-53db-4867-bd38-a8d45b15e40d","Type":"ContainerStarted","Data":"bdfda18f8d3fedd068b61a56faa49a42f63470547f20c02fe66f6dbeeaa64d13"} Oct 30 21:23:41.556539 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:41.556507 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:23:41.556708 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:41.556691 5748 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:23:42.190688 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:42.189148 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2r74x" Oct 30 21:23:42.201922 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:42.201869 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2r74x" Oct 30 21:23:42.287762 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:42.287733 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-75hkd" Oct 30 21:23:42.545996 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:42.545963 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fbd54bb44-d7zl5" event={"ID":"30b586ee-8d8a-4e95-b9a3-64f3b09c77e6","Type":"ContainerStarted","Data":"af535649f2985c0b00d326fada3f0ffa17d782627e119712f895c56b22fad00e"} Oct 30 21:23:43.809969 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:43.809924 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7fbd54bb44-d7zl5" Oct 30 21:23:44.560713 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:44.560675 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c","Type":"ContainerStarted","Data":"67a8e463bdb82a5f01f4e9d1b858150daf151747d36f12b8b75c03b4d7b97d97"} Oct 30 21:23:44.560891 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:44.560773 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c","Type":"ContainerStarted","Data":"182b0ff6179af260766d9d4e079c5b8318b9683165fcab661e728299b3378abd"} Oct 30 21:23:44.560891 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:44.560816 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c","Type":"ContainerStarted","Data":"18190e201ed740be7bc2134677ef1323cb3bdea8f6e830f26cc72e13e73861cc"} Oct 30 21:23:44.560891 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:44.560844 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c","Type":"ContainerStarted","Data":"d9e2491ee9698658b34ba43386367fe87f846e88042f26418ce0080b8cc14367"} Oct 30 21:23:44.560891 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:44.560873 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c","Type":"ContainerStarted","Data":"f3cb6d226c26bb7a3303eedbd832c79b10bb534328bf9859e8d7c6c875184010"} Oct 30 21:23:44.561079 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:44.560938 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c","Type":"ContainerStarted","Data":"c3595fa29b6875bba96aea742209bf807a364d5651724282cebc362cc1b99b58"} Oct 30 21:23:44.848324 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:44.847355 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.431027769 podStartE2EDuration="8.84733844s" totalImagesPullingTime="4.384765819s" totalInitContainerRuntime="31.544852ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:44.796843371 +0000 UTC m=+8.539364308" watchObservedRunningTime="2026-10-30 21:23:44.84733844 +0000 UTC m=+8.589859375" Oct 30 21:23:47.150106 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:47.150063 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7fbd54bb44-d7zl5" Oct 30 21:23:47.155802 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:47.155775 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7fbd54bb44-d7zl5" Oct 30 21:23:47.579677 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:47.579597 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7fbd54bb44-d7zl5" Oct 30 21:23:48.346573 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:48.346538 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:48.367159 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:48.367132 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:48.548343 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:48.548307 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:48.566572 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:48.566540 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:23:51.900400 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:51.900362 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j4v8r" Oct 30 21:23:54.930577 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:54.930548 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5dcb975b97-crpzf" Oct 30 21:23:55.061259 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:55.061217 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5dcb975b97-crpzf" Oct 30 21:23:55.898594 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:55.898562 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7fbd54bb44-d7zl5"] Oct 30 21:23:57.646287 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:57.646245 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-rq2vl" event={"ID":"2dbd490f-d94c-4f12-9106-a12117deda9d","Type":"ContainerStarted","Data":"af5470f109f24a4c58aea0a45e51b618e417113ba760bf87e501ab72a31524dd"} Oct 30 21:23:57.648453 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:57.648429 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-rq2vl" Oct 30 21:23:57.660540 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:23:57.660488 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-rq2vl" Oct 30 21:24:00.148351 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:24:00.148306 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:24:03.520305 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:03.520264 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 21:24:03.521211 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:03.521003 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1" containerName="kube-rbac-proxy" containerID="cri-o://3d284468d6c415172978473132a6e83ab9c2ecb67d9f97053b01fa3b62b871e0" gracePeriod=120 Oct 30 21:24:03.521211 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:03.521176 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1" containerName="alertmanager" containerID="cri-o://7297c874e6923540bf55507098ad23fe6dfda950450aa8d7b337c2721b4171d7" gracePeriod=120 Oct 30 21:24:03.521402 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:03.521043 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1" containerName="config-reloader" containerID="cri-o://da7a54e5c2563fc81cc86e63f88843464d1bd9606e0b7d6ffd0e8836db6677ef" gracePeriod=120 Oct 30 21:24:03.521402 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:03.521080 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1" containerName="prom-label-proxy" containerID="cri-o://1ab7380b3c86066a0ea469c93099103fcbe350a2670812a741bd38f032e50bd6" gracePeriod=120 Oct 30 21:24:03.521402 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:03.521090 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1" containerName="kube-rbac-proxy-web" containerID="cri-o://a0785c5b20f085f4118b42cdad2c706fe576eb55fe8dfc18165491e49e533545" gracePeriod=120 Oct 30 21:24:03.521556 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:03.521160 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1" containerName="kube-rbac-proxy-metric" containerID="cri-o://1839c3a83541f7d614e579209dc5e9561086789aa379aa32c22e64d32f1a3b57" gracePeriod=120 Oct 30 21:24:03.683532 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:03.683494 5748 generic.go:356] "Generic (PLEG): container finished" podID="d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1" containerID="1ab7380b3c86066a0ea469c93099103fcbe350a2670812a741bd38f032e50bd6" exitCode=0 Oct 30 21:24:03.683532 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:03.683528 5748 generic.go:356] "Generic (PLEG): container finished" podID="d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1" containerID="1839c3a83541f7d614e579209dc5e9561086789aa379aa32c22e64d32f1a3b57" exitCode=0 Oct 30 21:24:03.683726 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:03.683540 5748 generic.go:356] "Generic (PLEG): container finished" podID="d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1" containerID="3d284468d6c415172978473132a6e83ab9c2ecb67d9f97053b01fa3b62b871e0" exitCode=0 Oct 30 21:24:03.683726 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:03.683550 5748 generic.go:356] "Generic (PLEG): container finished" podID="d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1" containerID="a0785c5b20f085f4118b42cdad2c706fe576eb55fe8dfc18165491e49e533545" exitCode=0 Oct 30 21:24:03.683726 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:03.683555 5748 generic.go:356] "Generic (PLEG): container finished" podID="d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1" containerID="da7a54e5c2563fc81cc86e63f88843464d1bd9606e0b7d6ffd0e8836db6677ef" exitCode=0 Oct 30 21:24:03.683726 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:03.683562 5748 generic.go:356] "Generic (PLEG): container finished" podID="d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1" containerID="7297c874e6923540bf55507098ad23fe6dfda950450aa8d7b337c2721b4171d7" exitCode=0 Oct 30 21:24:03.683726 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:03.683571 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1","Type":"ContainerDied","Data":"1ab7380b3c86066a0ea469c93099103fcbe350a2670812a741bd38f032e50bd6"} Oct 30 21:24:03.683726 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:03.683685 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1","Type":"ContainerDied","Data":"1839c3a83541f7d614e579209dc5e9561086789aa379aa32c22e64d32f1a3b57"} Oct 30 21:24:03.683977 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:03.683731 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1","Type":"ContainerDied","Data":"3d284468d6c415172978473132a6e83ab9c2ecb67d9f97053b01fa3b62b871e0"} Oct 30 21:24:03.683977 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:03.683771 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1","Type":"ContainerDied","Data":"a0785c5b20f085f4118b42cdad2c706fe576eb55fe8dfc18165491e49e533545"} Oct 30 21:24:03.683977 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:03.683811 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1","Type":"ContainerDied","Data":"da7a54e5c2563fc81cc86e63f88843464d1bd9606e0b7d6ffd0e8836db6677ef"} Oct 30 21:24:03.683977 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:03.683839 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1","Type":"ContainerDied","Data":"7297c874e6923540bf55507098ad23fe6dfda950450aa8d7b337c2721b4171d7"} Oct 30 21:24:03.859521 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:03.859480 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:24:03.907609 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:03.907567 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-alertmanager-trusted-ca-bundle\") pod \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " Oct 30 21:24:03.907785 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:03.907646 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-metrics-client-ca\") pod \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " Oct 30 21:24:03.907785 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:03.907701 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " Oct 30 21:24:03.907785 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:03.907753 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-config-out\" (UniqueName: \"kubernetes.io/empty-dir/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-config-out\") pod \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " Oct 30 21:24:03.907998 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:03.907949 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-config-volume\" (UniqueName: \"kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-config-volume\") pod \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " Oct 30 21:24:03.908053 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:03.908023 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-alertmanager-main-db\") pod \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " Oct 30 21:24:03.908108 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:03.908074 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-secret-alertmanager-kube-rbac-proxy-web\") pod \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " Oct 30 21:24:03.908153 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:03.908104 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-alertmanager-trusted-ca-bundle" pod "d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1" (UID: "d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:24:03.908153 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:03.908113 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-metrics-client-ca" pod "d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1" (UID: "d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:24:03.908153 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:03.908112 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-cluster-tls-config\") pod \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " Oct 30 21:24:03.908303 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:03.908193 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-web-config\" (UniqueName: \"kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-web-config\") pod \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " Oct 30 21:24:03.908303 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:03.908236 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-secret-alertmanager-kube-rbac-proxy\") pod \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " Oct 30 21:24:03.908303 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:03.908272 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-kube-api-access-sw9tn\" (UniqueName: \"kubernetes.io/projected/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-kube-api-access-sw9tn\") pod \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " Oct 30 21:24:03.908303 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:03.908299 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-tls-assets\" (UniqueName: \"kubernetes.io/projected/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-tls-assets\") pod \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " Oct 30 21:24:03.908498 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:03.908397 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-alertmanager-main-db" pod "d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1" (UID: "d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:24:03.908498 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:03.908475 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-secret-alertmanager-main-tls\") pod \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\" (UID: \"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1\") " Oct 30 21:24:03.909103 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:03.908782 5748 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:03.909103 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:03.908805 5748 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-metrics-client-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:03.909103 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:03.908821 5748 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-alertmanager-main-db\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:03.912986 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:03.912955 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-tls-assets" pod "d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1" (UID: "d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:24:03.913164 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:03.913006 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-secret-alertmanager-kube-rbac-proxy-web" pod "d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1" (UID: "d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:24:03.914286 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:03.914258 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-secret-alertmanager-kube-rbac-proxy-metric" pod "d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1" (UID: "d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:24:03.914479 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:03.914394 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-secret-alertmanager-main-tls" pod "d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1" (UID: "d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:24:03.914699 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:03.914670 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-secret-alertmanager-kube-rbac-proxy" pod "d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1" (UID: "d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:24:03.914831 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:03.914723 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-config-out" pod "d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1" (UID: "d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:24:03.914831 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:03.914819 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-config-volume" pod "d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1" (UID: "d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:24:03.916281 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:03.916253 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-kube-api-access-sw9tn" pod "d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1" (UID: "d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1"). InnerVolumeSpecName "kube-api-access-sw9tn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:24:03.919065 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:03.919036 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-cluster-tls-config" pod "d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1" (UID: "d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:24:03.928063 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:03.928029 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-web-config" pod "d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1" (UID: "d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:24:04.009717 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:04.009643 5748 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-cluster-tls-config\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:04.009717 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:04.009719 5748 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-web-config\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:04.010018 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:04.009738 5748 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:04.010018 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:04.009754 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sw9tn\" (UniqueName: \"kubernetes.io/projected/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-kube-api-access-sw9tn\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:04.010018 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:04.009767 5748 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-tls-assets\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:04.010018 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:04.009823 5748 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-secret-alertmanager-main-tls\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:04.010018 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:04.009841 5748 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:04.010018 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:04.009857 5748 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-config-out\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:04.010018 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:04.009871 5748 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-config-volume\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:04.010018 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:04.009884 5748 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:04.691998 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:04.691922 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:24:04.692445 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:04.691913 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1","Type":"ContainerDied","Data":"ac3b2f70532aadcd84f622494b52343b96ea6fa8e52d777db57d766979f199fb"} Oct 30 21:24:04.692445 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:04.692099 5748 scope.go:122] "RemoveContainer" containerID="1ab7380b3c86066a0ea469c93099103fcbe350a2670812a741bd38f032e50bd6" Oct 30 21:24:04.701356 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:04.701328 5748 scope.go:122] "RemoveContainer" containerID="1839c3a83541f7d614e579209dc5e9561086789aa379aa32c22e64d32f1a3b57" Oct 30 21:24:04.710074 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:04.710048 5748 scope.go:122] "RemoveContainer" containerID="3d284468d6c415172978473132a6e83ab9c2ecb67d9f97053b01fa3b62b871e0" Oct 30 21:24:04.718226 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:04.718135 5748 scope.go:122] "RemoveContainer" containerID="a0785c5b20f085f4118b42cdad2c706fe576eb55fe8dfc18165491e49e533545" Oct 30 21:24:04.726379 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:04.726355 5748 scope.go:122] "RemoveContainer" containerID="da7a54e5c2563fc81cc86e63f88843464d1bd9606e0b7d6ffd0e8836db6677ef" Oct 30 21:24:04.735288 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:04.735266 5748 scope.go:122] "RemoveContainer" containerID="7297c874e6923540bf55507098ad23fe6dfda950450aa8d7b337c2721b4171d7" Oct 30 21:24:04.744072 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:04.744047 5748 scope.go:122] "RemoveContainer" containerID="ffef4be78f8c37e05b78d506ffd53a0b58cd89c523b3cab54930fca37f7ccc6a" Oct 30 21:24:04.862838 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:04.862795 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 21:24:04.912973 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:04.912940 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 21:24:06.414994 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:06.414960 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1" path="/var/lib/kubelet/pods/d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1/volumes" Oct 30 21:24:09.309432 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:09.309379 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:24:09.310137 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:09.310074 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bce4cfcf-5926-433e-a0dd-238dfbb6ef1c" containerName="prometheus" containerID="cri-o://c3595fa29b6875bba96aea742209bf807a364d5651724282cebc362cc1b99b58" gracePeriod=600 Oct 30 21:24:09.310284 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:09.310123 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bce4cfcf-5926-433e-a0dd-238dfbb6ef1c" containerName="kube-rbac-proxy-web" containerID="cri-o://18190e201ed740be7bc2134677ef1323cb3bdea8f6e830f26cc72e13e73861cc" gracePeriod=600 Oct 30 21:24:09.310284 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:09.310199 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bce4cfcf-5926-433e-a0dd-238dfbb6ef1c" containerName="kube-rbac-proxy-thanos" containerID="cri-o://67a8e463bdb82a5f01f4e9d1b858150daf151747d36f12b8b75c03b4d7b97d97" gracePeriod=600 Oct 30 21:24:09.310284 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:09.310132 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bce4cfcf-5926-433e-a0dd-238dfbb6ef1c" containerName="thanos-sidecar" containerID="cri-o://d9e2491ee9698658b34ba43386367fe87f846e88042f26418ce0080b8cc14367" gracePeriod=600 Oct 30 21:24:09.310446 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:09.310281 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bce4cfcf-5926-433e-a0dd-238dfbb6ef1c" containerName="config-reloader" containerID="cri-o://f3cb6d226c26bb7a3303eedbd832c79b10bb534328bf9859e8d7c6c875184010" gracePeriod=600 Oct 30 21:24:09.310446 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:09.310280 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bce4cfcf-5926-433e-a0dd-238dfbb6ef1c" containerName="kube-rbac-proxy" containerID="cri-o://182b0ff6179af260766d9d4e079c5b8318b9683165fcab661e728299b3378abd" gracePeriod=600 Oct 30 21:24:09.704118 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:09.704085 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-wv464" Oct 30 21:24:09.714083 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:09.714055 5748 generic.go:356] "Generic (PLEG): container finished" podID="bce4cfcf-5926-433e-a0dd-238dfbb6ef1c" containerID="67a8e463bdb82a5f01f4e9d1b858150daf151747d36f12b8b75c03b4d7b97d97" exitCode=0 Oct 30 21:24:09.714083 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:09.714081 5748 generic.go:356] "Generic (PLEG): container finished" podID="bce4cfcf-5926-433e-a0dd-238dfbb6ef1c" containerID="182b0ff6179af260766d9d4e079c5b8318b9683165fcab661e728299b3378abd" exitCode=0 Oct 30 21:24:09.714289 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:09.714091 5748 generic.go:356] "Generic (PLEG): container finished" podID="bce4cfcf-5926-433e-a0dd-238dfbb6ef1c" containerID="d9e2491ee9698658b34ba43386367fe87f846e88042f26418ce0080b8cc14367" exitCode=0 Oct 30 21:24:09.714289 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:09.714101 5748 generic.go:356] "Generic (PLEG): container finished" podID="bce4cfcf-5926-433e-a0dd-238dfbb6ef1c" containerID="f3cb6d226c26bb7a3303eedbd832c79b10bb534328bf9859e8d7c6c875184010" exitCode=0 Oct 30 21:24:09.714289 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:09.714111 5748 generic.go:356] "Generic (PLEG): container finished" podID="bce4cfcf-5926-433e-a0dd-238dfbb6ef1c" containerID="c3595fa29b6875bba96aea742209bf807a364d5651724282cebc362cc1b99b58" exitCode=0 Oct 30 21:24:09.714289 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:09.714127 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c","Type":"ContainerDied","Data":"67a8e463bdb82a5f01f4e9d1b858150daf151747d36f12b8b75c03b4d7b97d97"} Oct 30 21:24:09.714289 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:09.714203 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c","Type":"ContainerDied","Data":"182b0ff6179af260766d9d4e079c5b8318b9683165fcab661e728299b3378abd"} Oct 30 21:24:09.714289 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:09.714237 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c","Type":"ContainerDied","Data":"d9e2491ee9698658b34ba43386367fe87f846e88042f26418ce0080b8cc14367"} Oct 30 21:24:09.714289 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:09.714264 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c","Type":"ContainerDied","Data":"f3cb6d226c26bb7a3303eedbd832c79b10bb534328bf9859e8d7c6c875184010"} Oct 30 21:24:09.714289 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:09.714287 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c","Type":"ContainerDied","Data":"c3595fa29b6875bba96aea742209bf807a364d5651724282cebc362cc1b99b58"} Oct 30 21:24:10.584461 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.584427 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:10.669205 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.669111 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-prometheus-trusted-ca-bundle\") pod \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " Oct 30 21:24:10.669205 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.669154 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-thanos-prometheus-http-client-file\") pod \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " Oct 30 21:24:10.669205 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.669183 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-tls-assets\" (UniqueName: \"kubernetes.io/projected/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-tls-assets\") pod \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " Oct 30 21:24:10.669484 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.669228 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " Oct 30 21:24:10.669484 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.669253 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-prometheus-k8s-rulefiles-0\") pod \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " Oct 30 21:24:10.669484 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.669287 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-kube-api-access-jngrf\" (UniqueName: \"kubernetes.io/projected/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-kube-api-access-jngrf\") pod \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " Oct 30 21:24:10.669484 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.669321 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " Oct 30 21:24:10.669484 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.669352 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-config-out\" (UniqueName: \"kubernetes.io/empty-dir/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-config-out\") pod \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " Oct 30 21:24:10.669484 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.669394 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-secret-grpc-tls\") pod \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " Oct 30 21:24:10.669484 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.669437 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-web-config\" (UniqueName: \"kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-web-config\") pod \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " Oct 30 21:24:10.669484 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.669466 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-configmap-serving-certs-ca-bundle\") pod \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " Oct 30 21:24:10.670237 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.669503 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-secret-kube-rbac-proxy\") pod \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " Oct 30 21:24:10.670237 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.669557 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-prometheus-k8s-db\") pod \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " Oct 30 21:24:10.670237 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.669591 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-configmap-kubelet-serving-ca-bundle\") pod \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " Oct 30 21:24:10.670237 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.669593 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-prometheus-trusted-ca-bundle" pod "bce4cfcf-5926-433e-a0dd-238dfbb6ef1c" (UID: "bce4cfcf-5926-433e-a0dd-238dfbb6ef1c"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:24:10.670237 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.669620 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-secret-prometheus-k8s-tls\") pod \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " Oct 30 21:24:10.670237 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.669647 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-configmap-metrics-client-ca\") pod \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " Oct 30 21:24:10.670237 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.669695 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-prometheus-k8s-rulefiles-1\") pod \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " Oct 30 21:24:10.670237 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.669738 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-config\" (UniqueName: \"kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-config\") pod \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " Oct 30 21:24:10.670237 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.669766 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-secret-metrics-client-certs\") pod \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " Oct 30 21:24:10.670237 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.669810 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-prometheus-k8s-rulefiles-2\") pod \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\" (UID: \"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c\") " Oct 30 21:24:10.670237 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.670125 5748 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-prometheus-trusted-ca-bundle\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:10.670818 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.670461 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-prometheus-k8s-rulefiles-2" pod "bce4cfcf-5926-433e-a0dd-238dfbb6ef1c" (UID: "bce4cfcf-5926-433e-a0dd-238dfbb6ef1c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:24:10.670818 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.670531 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-configmap-serving-certs-ca-bundle" pod "bce4cfcf-5926-433e-a0dd-238dfbb6ef1c" (UID: "bce4cfcf-5926-433e-a0dd-238dfbb6ef1c"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:24:10.670818 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.670763 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-prometheus-k8s-rulefiles-1" pod "bce4cfcf-5926-433e-a0dd-238dfbb6ef1c" (UID: "bce4cfcf-5926-433e-a0dd-238dfbb6ef1c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:24:10.671163 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.671129 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-configmap-metrics-client-ca" pod "bce4cfcf-5926-433e-a0dd-238dfbb6ef1c" (UID: "bce4cfcf-5926-433e-a0dd-238dfbb6ef1c"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:24:10.672158 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.672127 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-prometheus-k8s-db" pod "bce4cfcf-5926-433e-a0dd-238dfbb6ef1c" (UID: "bce4cfcf-5926-433e-a0dd-238dfbb6ef1c"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:24:10.672722 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.672582 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-prometheus-k8s-rulefiles-0" pod "bce4cfcf-5926-433e-a0dd-238dfbb6ef1c" (UID: "bce4cfcf-5926-433e-a0dd-238dfbb6ef1c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:24:10.672722 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.672668 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-thanos-prometheus-http-client-file" pod "bce4cfcf-5926-433e-a0dd-238dfbb6ef1c" (UID: "bce4cfcf-5926-433e-a0dd-238dfbb6ef1c"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:24:10.673212 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.673091 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-configmap-kubelet-serving-ca-bundle" pod "bce4cfcf-5926-433e-a0dd-238dfbb6ef1c" (UID: "bce4cfcf-5926-433e-a0dd-238dfbb6ef1c"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:24:10.673212 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.673143 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-tls-assets" pod "bce4cfcf-5926-433e-a0dd-238dfbb6ef1c" (UID: "bce4cfcf-5926-433e-a0dd-238dfbb6ef1c"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:24:10.673378 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.673213 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-secret-prometheus-k8s-thanos-sidecar-tls" pod "bce4cfcf-5926-433e-a0dd-238dfbb6ef1c" (UID: "bce4cfcf-5926-433e-a0dd-238dfbb6ef1c"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:24:10.673641 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.673616 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-kube-api-access-jngrf" pod "bce4cfcf-5926-433e-a0dd-238dfbb6ef1c" (UID: "bce4cfcf-5926-433e-a0dd-238dfbb6ef1c"). InnerVolumeSpecName "kube-api-access-jngrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:24:10.673823 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.673804 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-secret-metrics-client-certs" pod "bce4cfcf-5926-433e-a0dd-238dfbb6ef1c" (UID: "bce4cfcf-5926-433e-a0dd-238dfbb6ef1c"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:24:10.673891 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.673820 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-config-out" pod "bce4cfcf-5926-433e-a0dd-238dfbb6ef1c" (UID: "bce4cfcf-5926-433e-a0dd-238dfbb6ef1c"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:24:10.674001 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.673976 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-secret-prometheus-k8s-kube-rbac-proxy-web" pod "bce4cfcf-5926-433e-a0dd-238dfbb6ef1c" (UID: "bce4cfcf-5926-433e-a0dd-238dfbb6ef1c"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:24:10.674231 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.674202 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-secret-prometheus-k8s-tls" pod "bce4cfcf-5926-433e-a0dd-238dfbb6ef1c" (UID: "bce4cfcf-5926-433e-a0dd-238dfbb6ef1c"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:24:10.675312 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.675286 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-secret-grpc-tls" pod "bce4cfcf-5926-433e-a0dd-238dfbb6ef1c" (UID: "bce4cfcf-5926-433e-a0dd-238dfbb6ef1c"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:24:10.675385 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.675361 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-config" pod "bce4cfcf-5926-433e-a0dd-238dfbb6ef1c" (UID: "bce4cfcf-5926-433e-a0dd-238dfbb6ef1c"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:24:10.675486 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.675466 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-secret-kube-rbac-proxy" pod "bce4cfcf-5926-433e-a0dd-238dfbb6ef1c" (UID: "bce4cfcf-5926-433e-a0dd-238dfbb6ef1c"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:24:10.685038 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.685008 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-web-config" pod "bce4cfcf-5926-433e-a0dd-238dfbb6ef1c" (UID: "bce4cfcf-5926-433e-a0dd-238dfbb6ef1c"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:24:10.721084 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.721046 5748 generic.go:356] "Generic (PLEG): container finished" podID="bce4cfcf-5926-433e-a0dd-238dfbb6ef1c" containerID="18190e201ed740be7bc2134677ef1323cb3bdea8f6e830f26cc72e13e73861cc" exitCode=0 Oct 30 21:24:10.721245 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.721091 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c","Type":"ContainerDied","Data":"18190e201ed740be7bc2134677ef1323cb3bdea8f6e830f26cc72e13e73861cc"} Oct 30 21:24:10.721245 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.721142 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:10.721245 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.721164 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bce4cfcf-5926-433e-a0dd-238dfbb6ef1c","Type":"ContainerDied","Data":"5a5ce108388966ac477aac72c60f47a51638b2b20925a37abb6f5235903187fc"} Oct 30 21:24:10.721245 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.721211 5748 scope.go:122] "RemoveContainer" containerID="67a8e463bdb82a5f01f4e9d1b858150daf151747d36f12b8b75c03b4d7b97d97" Oct 30 21:24:10.729425 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.729405 5748 scope.go:122] "RemoveContainer" containerID="182b0ff6179af260766d9d4e079c5b8318b9683165fcab661e728299b3378abd" Oct 30 21:24:10.736397 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.736380 5748 scope.go:122] "RemoveContainer" containerID="18190e201ed740be7bc2134677ef1323cb3bdea8f6e830f26cc72e13e73861cc" Oct 30 21:24:10.743261 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.743229 5748 scope.go:122] "RemoveContainer" containerID="d9e2491ee9698658b34ba43386367fe87f846e88042f26418ce0080b8cc14367" Oct 30 21:24:10.750069 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.750048 5748 scope.go:122] "RemoveContainer" containerID="f3cb6d226c26bb7a3303eedbd832c79b10bb534328bf9859e8d7c6c875184010" Oct 30 21:24:10.756770 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.756750 5748 scope.go:122] "RemoveContainer" containerID="c3595fa29b6875bba96aea742209bf807a364d5651724282cebc362cc1b99b58" Oct 30 21:24:10.763763 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.763745 5748 scope.go:122] "RemoveContainer" containerID="afe6661de93bfc5e26c209186b3db8d4b82dbe82485e926e05bf19ad430201de" Oct 30 21:24:10.770584 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.770565 5748 scope.go:122] "RemoveContainer" containerID="67a8e463bdb82a5f01f4e9d1b858150daf151747d36f12b8b75c03b4d7b97d97" Oct 30 21:24:10.770758 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.770737 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jngrf\" (UniqueName: \"kubernetes.io/projected/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-kube-api-access-jngrf\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:10.770825 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.770767 5748 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:10.770825 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.770784 5748 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-config-out\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:10.770825 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.770799 5748 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-secret-grpc-tls\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:10.770825 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.770814 5748 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-web-config\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:10.771191 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.770832 5748 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:10.771191 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.770846 5748 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-secret-kube-rbac-proxy\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:10.771191 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:24:10.770850 5748 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67a8e463bdb82a5f01f4e9d1b858150daf151747d36f12b8b75c03b4d7b97d97\": container with ID starting with 67a8e463bdb82a5f01f4e9d1b858150daf151747d36f12b8b75c03b4d7b97d97 not found: ID does not exist" containerID="67a8e463bdb82a5f01f4e9d1b858150daf151747d36f12b8b75c03b4d7b97d97" Oct 30 21:24:10.771191 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.770862 5748 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-prometheus-k8s-db\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:10.771191 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.770878 5748 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:10.771191 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.770906 5748 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-secret-prometheus-k8s-tls\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:10.771191 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.770877 5748 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67a8e463bdb82a5f01f4e9d1b858150daf151747d36f12b8b75c03b4d7b97d97"} err="failed to get container status \"67a8e463bdb82a5f01f4e9d1b858150daf151747d36f12b8b75c03b4d7b97d97\": rpc error: code = NotFound desc = could not find container \"67a8e463bdb82a5f01f4e9d1b858150daf151747d36f12b8b75c03b4d7b97d97\": container with ID starting with 67a8e463bdb82a5f01f4e9d1b858150daf151747d36f12b8b75c03b4d7b97d97 not found: ID does not exist" Oct 30 21:24:10.771191 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.770932 5748 scope.go:122] "RemoveContainer" containerID="182b0ff6179af260766d9d4e079c5b8318b9683165fcab661e728299b3378abd" Oct 30 21:24:10.771191 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.770921 5748 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-configmap-metrics-client-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:10.771191 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.770990 5748 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:10.771191 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.771003 5748 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-config\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:10.771191 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.771017 5748 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-secret-metrics-client-certs\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:10.771191 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.771033 5748 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:10.771191 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.771044 5748 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-thanos-prometheus-http-client-file\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:10.771191 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.771053 5748 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-tls-assets\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:10.771191 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.771063 5748 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:10.771191 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.771075 5748 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:10.771724 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:24:10.771209 5748 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"182b0ff6179af260766d9d4e079c5b8318b9683165fcab661e728299b3378abd\": container with ID starting with 182b0ff6179af260766d9d4e079c5b8318b9683165fcab661e728299b3378abd not found: ID does not exist" containerID="182b0ff6179af260766d9d4e079c5b8318b9683165fcab661e728299b3378abd" Oct 30 21:24:10.771724 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.771239 5748 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"182b0ff6179af260766d9d4e079c5b8318b9683165fcab661e728299b3378abd"} err="failed to get container status \"182b0ff6179af260766d9d4e079c5b8318b9683165fcab661e728299b3378abd\": rpc error: code = NotFound desc = could not find container \"182b0ff6179af260766d9d4e079c5b8318b9683165fcab661e728299b3378abd\": container with ID starting with 182b0ff6179af260766d9d4e079c5b8318b9683165fcab661e728299b3378abd not found: ID does not exist" Oct 30 21:24:10.771724 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.771255 5748 scope.go:122] "RemoveContainer" containerID="18190e201ed740be7bc2134677ef1323cb3bdea8f6e830f26cc72e13e73861cc" Oct 30 21:24:10.771724 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:24:10.771480 5748 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18190e201ed740be7bc2134677ef1323cb3bdea8f6e830f26cc72e13e73861cc\": container with ID starting with 18190e201ed740be7bc2134677ef1323cb3bdea8f6e830f26cc72e13e73861cc not found: ID does not exist" containerID="18190e201ed740be7bc2134677ef1323cb3bdea8f6e830f26cc72e13e73861cc" Oct 30 21:24:10.771724 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.771501 5748 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18190e201ed740be7bc2134677ef1323cb3bdea8f6e830f26cc72e13e73861cc"} err="failed to get container status \"18190e201ed740be7bc2134677ef1323cb3bdea8f6e830f26cc72e13e73861cc\": rpc error: code = NotFound desc = could not find container \"18190e201ed740be7bc2134677ef1323cb3bdea8f6e830f26cc72e13e73861cc\": container with ID starting with 18190e201ed740be7bc2134677ef1323cb3bdea8f6e830f26cc72e13e73861cc not found: ID does not exist" Oct 30 21:24:10.771724 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.771516 5748 scope.go:122] "RemoveContainer" containerID="d9e2491ee9698658b34ba43386367fe87f846e88042f26418ce0080b8cc14367" Oct 30 21:24:10.771724 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:24:10.771719 5748 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9e2491ee9698658b34ba43386367fe87f846e88042f26418ce0080b8cc14367\": container with ID starting with d9e2491ee9698658b34ba43386367fe87f846e88042f26418ce0080b8cc14367 not found: ID does not exist" containerID="d9e2491ee9698658b34ba43386367fe87f846e88042f26418ce0080b8cc14367" Oct 30 21:24:10.771994 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.771736 5748 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9e2491ee9698658b34ba43386367fe87f846e88042f26418ce0080b8cc14367"} err="failed to get container status \"d9e2491ee9698658b34ba43386367fe87f846e88042f26418ce0080b8cc14367\": rpc error: code = NotFound desc = could not find container \"d9e2491ee9698658b34ba43386367fe87f846e88042f26418ce0080b8cc14367\": container with ID starting with d9e2491ee9698658b34ba43386367fe87f846e88042f26418ce0080b8cc14367 not found: ID does not exist" Oct 30 21:24:10.771994 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.771767 5748 scope.go:122] "RemoveContainer" containerID="f3cb6d226c26bb7a3303eedbd832c79b10bb534328bf9859e8d7c6c875184010" Oct 30 21:24:10.771994 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:24:10.771986 5748 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3cb6d226c26bb7a3303eedbd832c79b10bb534328bf9859e8d7c6c875184010\": container with ID starting with f3cb6d226c26bb7a3303eedbd832c79b10bb534328bf9859e8d7c6c875184010 not found: ID does not exist" containerID="f3cb6d226c26bb7a3303eedbd832c79b10bb534328bf9859e8d7c6c875184010" Oct 30 21:24:10.772085 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.772005 5748 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3cb6d226c26bb7a3303eedbd832c79b10bb534328bf9859e8d7c6c875184010"} err="failed to get container status \"f3cb6d226c26bb7a3303eedbd832c79b10bb534328bf9859e8d7c6c875184010\": rpc error: code = NotFound desc = could not find container \"f3cb6d226c26bb7a3303eedbd832c79b10bb534328bf9859e8d7c6c875184010\": container with ID starting with f3cb6d226c26bb7a3303eedbd832c79b10bb534328bf9859e8d7c6c875184010 not found: ID does not exist" Oct 30 21:24:10.772085 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.772019 5748 scope.go:122] "RemoveContainer" containerID="c3595fa29b6875bba96aea742209bf807a364d5651724282cebc362cc1b99b58" Oct 30 21:24:10.772214 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:24:10.772194 5748 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3595fa29b6875bba96aea742209bf807a364d5651724282cebc362cc1b99b58\": container with ID starting with c3595fa29b6875bba96aea742209bf807a364d5651724282cebc362cc1b99b58 not found: ID does not exist" containerID="c3595fa29b6875bba96aea742209bf807a364d5651724282cebc362cc1b99b58" Oct 30 21:24:10.772252 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.772218 5748 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3595fa29b6875bba96aea742209bf807a364d5651724282cebc362cc1b99b58"} err="failed to get container status \"c3595fa29b6875bba96aea742209bf807a364d5651724282cebc362cc1b99b58\": rpc error: code = NotFound desc = could not find container \"c3595fa29b6875bba96aea742209bf807a364d5651724282cebc362cc1b99b58\": container with ID starting with c3595fa29b6875bba96aea742209bf807a364d5651724282cebc362cc1b99b58 not found: ID does not exist" Oct 30 21:24:10.772252 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.772230 5748 scope.go:122] "RemoveContainer" containerID="afe6661de93bfc5e26c209186b3db8d4b82dbe82485e926e05bf19ad430201de" Oct 30 21:24:10.772428 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:24:10.772409 5748 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afe6661de93bfc5e26c209186b3db8d4b82dbe82485e926e05bf19ad430201de\": container with ID starting with afe6661de93bfc5e26c209186b3db8d4b82dbe82485e926e05bf19ad430201de not found: ID does not exist" containerID="afe6661de93bfc5e26c209186b3db8d4b82dbe82485e926e05bf19ad430201de" Oct 30 21:24:10.772467 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.772431 5748 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afe6661de93bfc5e26c209186b3db8d4b82dbe82485e926e05bf19ad430201de"} err="failed to get container status \"afe6661de93bfc5e26c209186b3db8d4b82dbe82485e926e05bf19ad430201de\": rpc error: code = NotFound desc = could not find container \"afe6661de93bfc5e26c209186b3db8d4b82dbe82485e926e05bf19ad430201de\": container with ID starting with afe6661de93bfc5e26c209186b3db8d4b82dbe82485e926e05bf19ad430201de not found: ID does not exist" Oct 30 21:24:10.957817 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.957734 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:24:10.963910 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.963867 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:24:10.995182 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.995150 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:24:10.995456 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.995443 5748 cpu_manager.go:414] "containerMap: removing container" podUID="bce4cfcf-5926-433e-a0dd-238dfbb6ef1c" containerName="kube-rbac-proxy-web" Oct 30 21:24:10.995501 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.995457 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bce4cfcf-5926-433e-a0dd-238dfbb6ef1c" containerName="kube-rbac-proxy-web" Oct 30 21:24:10.995501 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.995471 5748 cpu_manager.go:414] "containerMap: removing container" podUID="bce4cfcf-5926-433e-a0dd-238dfbb6ef1c" containerName="thanos-sidecar" Oct 30 21:24:10.995501 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.995476 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bce4cfcf-5926-433e-a0dd-238dfbb6ef1c" containerName="thanos-sidecar" Oct 30 21:24:10.995501 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.995488 5748 cpu_manager.go:414] "containerMap: removing container" podUID="bce4cfcf-5926-433e-a0dd-238dfbb6ef1c" containerName="init-config-reloader" Oct 30 21:24:10.995501 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.995493 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bce4cfcf-5926-433e-a0dd-238dfbb6ef1c" containerName="init-config-reloader" Oct 30 21:24:10.995501 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.995501 5748 cpu_manager.go:414] "containerMap: removing container" podUID="d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1" containerName="prom-label-proxy" Oct 30 21:24:10.995715 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.995506 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1" containerName="prom-label-proxy" Oct 30 21:24:10.995715 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.995512 5748 cpu_manager.go:414] "containerMap: removing container" podUID="bce4cfcf-5926-433e-a0dd-238dfbb6ef1c" containerName="config-reloader" Oct 30 21:24:10.995715 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.995518 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bce4cfcf-5926-433e-a0dd-238dfbb6ef1c" containerName="config-reloader" Oct 30 21:24:10.995715 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.995528 5748 cpu_manager.go:414] "containerMap: removing container" podUID="d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1" containerName="kube-rbac-proxy" Oct 30 21:24:10.995715 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.995532 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1" containerName="kube-rbac-proxy" Oct 30 21:24:10.995715 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.995542 5748 cpu_manager.go:414] "containerMap: removing container" podUID="bce4cfcf-5926-433e-a0dd-238dfbb6ef1c" containerName="prometheus" Oct 30 21:24:10.995715 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.995546 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bce4cfcf-5926-433e-a0dd-238dfbb6ef1c" containerName="prometheus" Oct 30 21:24:10.995715 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.995552 5748 cpu_manager.go:414] "containerMap: removing container" podUID="bce4cfcf-5926-433e-a0dd-238dfbb6ef1c" containerName="kube-rbac-proxy-thanos" Oct 30 21:24:10.995715 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.995556 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bce4cfcf-5926-433e-a0dd-238dfbb6ef1c" containerName="kube-rbac-proxy-thanos" Oct 30 21:24:10.995715 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.995563 5748 cpu_manager.go:414] "containerMap: removing container" podUID="d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1" containerName="config-reloader" Oct 30 21:24:10.995715 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.995567 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1" containerName="config-reloader" Oct 30 21:24:10.995715 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.995575 5748 cpu_manager.go:414] "containerMap: removing container" podUID="d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1" containerName="alertmanager" Oct 30 21:24:10.995715 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.995579 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1" containerName="alertmanager" Oct 30 21:24:10.995715 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.995586 5748 cpu_manager.go:414] "containerMap: removing container" podUID="d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1" containerName="kube-rbac-proxy-web" Oct 30 21:24:10.995715 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.995590 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1" containerName="kube-rbac-proxy-web" Oct 30 21:24:10.995715 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.995596 5748 cpu_manager.go:414] "containerMap: removing container" podUID="d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1" containerName="init-config-reloader" Oct 30 21:24:10.995715 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.995601 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1" containerName="init-config-reloader" Oct 30 21:24:10.995715 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.995610 5748 cpu_manager.go:414] "containerMap: removing container" podUID="d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1" containerName="kube-rbac-proxy-metric" Oct 30 21:24:10.995715 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.995614 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1" containerName="kube-rbac-proxy-metric" Oct 30 21:24:10.995715 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.995620 5748 cpu_manager.go:414] "containerMap: removing container" podUID="bce4cfcf-5926-433e-a0dd-238dfbb6ef1c" containerName="kube-rbac-proxy" Oct 30 21:24:10.995715 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.995624 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bce4cfcf-5926-433e-a0dd-238dfbb6ef1c" containerName="kube-rbac-proxy" Oct 30 21:24:10.995715 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.995667 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1" containerName="prom-label-proxy" Oct 30 21:24:10.995715 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.995674 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="bce4cfcf-5926-433e-a0dd-238dfbb6ef1c" containerName="config-reloader" Oct 30 21:24:10.995715 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.995681 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1" containerName="kube-rbac-proxy" Oct 30 21:24:10.995715 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.995687 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="bce4cfcf-5926-433e-a0dd-238dfbb6ef1c" containerName="prometheus" Oct 30 21:24:10.995715 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.995693 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="bce4cfcf-5926-433e-a0dd-238dfbb6ef1c" containerName="kube-rbac-proxy-thanos" Oct 30 21:24:10.995715 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.995698 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1" containerName="config-reloader" Oct 30 21:24:10.995715 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.995704 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1" containerName="alertmanager" Oct 30 21:24:10.995715 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.995710 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1" containerName="kube-rbac-proxy-web" Oct 30 21:24:10.995715 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.995715 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="d96c9c66-7bd6-4ee6-b6e5-1ca97b1dccc1" containerName="kube-rbac-proxy-metric" Oct 30 21:24:10.995715 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.995721 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="bce4cfcf-5926-433e-a0dd-238dfbb6ef1c" containerName="kube-rbac-proxy" Oct 30 21:24:10.995715 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.995727 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="bce4cfcf-5926-433e-a0dd-238dfbb6ef1c" containerName="kube-rbac-proxy-web" Oct 30 21:24:10.996621 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:10.995733 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="bce4cfcf-5926-433e-a0dd-238dfbb6ef1c" containerName="thanos-sidecar" Oct 30 21:24:11.019460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.019431 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:11.073455 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.073422 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b287c83b-aa37-4063-9782-b62b202832d5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b287c83b-aa37-4063-9782-b62b202832d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:11.073629 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.073459 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b287c83b-aa37-4063-9782-b62b202832d5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b287c83b-aa37-4063-9782-b62b202832d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:11.073629 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.073539 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b287c83b-aa37-4063-9782-b62b202832d5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b287c83b-aa37-4063-9782-b62b202832d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:11.073629 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.073576 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b287c83b-aa37-4063-9782-b62b202832d5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b287c83b-aa37-4063-9782-b62b202832d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:11.073629 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.073612 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b287c83b-aa37-4063-9782-b62b202832d5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b287c83b-aa37-4063-9782-b62b202832d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:11.073795 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.073716 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b287c83b-aa37-4063-9782-b62b202832d5-config-out\") pod \"prometheus-k8s-0\" (UID: \"b287c83b-aa37-4063-9782-b62b202832d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:11.073795 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.073755 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b287c83b-aa37-4063-9782-b62b202832d5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b287c83b-aa37-4063-9782-b62b202832d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:11.073795 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.073775 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b287c83b-aa37-4063-9782-b62b202832d5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b287c83b-aa37-4063-9782-b62b202832d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:11.073928 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.073798 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b287c83b-aa37-4063-9782-b62b202832d5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b287c83b-aa37-4063-9782-b62b202832d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:11.073928 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.073852 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b287c83b-aa37-4063-9782-b62b202832d5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b287c83b-aa37-4063-9782-b62b202832d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:11.073928 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.073884 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b287c83b-aa37-4063-9782-b62b202832d5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b287c83b-aa37-4063-9782-b62b202832d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:11.074100 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.073938 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b287c83b-aa37-4063-9782-b62b202832d5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b287c83b-aa37-4063-9782-b62b202832d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:11.074100 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.073965 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b287c83b-aa37-4063-9782-b62b202832d5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b287c83b-aa37-4063-9782-b62b202832d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:11.074100 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.073985 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-2sgxh\"" Oct 30 21:24:11.074100 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.073986 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 21:24:11.074100 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.073986 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 21:24:11.074100 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.074061 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b287c83b-aa37-4063-9782-b62b202832d5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b287c83b-aa37-4063-9782-b62b202832d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:11.074100 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.074089 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b287c83b-aa37-4063-9782-b62b202832d5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b287c83b-aa37-4063-9782-b62b202832d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:11.074306 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.074105 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7cgl\" (UniqueName: \"kubernetes.io/projected/b287c83b-aa37-4063-9782-b62b202832d5-kube-api-access-c7cgl\") pod \"prometheus-k8s-0\" (UID: \"b287c83b-aa37-4063-9782-b62b202832d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:11.074306 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.074179 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b287c83b-aa37-4063-9782-b62b202832d5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b287c83b-aa37-4063-9782-b62b202832d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:11.074306 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.074208 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b287c83b-aa37-4063-9782-b62b202832d5-config\") pod \"prometheus-k8s-0\" (UID: \"b287c83b-aa37-4063-9782-b62b202832d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:11.074306 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.074225 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b287c83b-aa37-4063-9782-b62b202832d5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b287c83b-aa37-4063-9782-b62b202832d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:11.074306 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.074245 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b287c83b-aa37-4063-9782-b62b202832d5-web-config\") pod \"prometheus-k8s-0\" (UID: \"b287c83b-aa37-4063-9782-b62b202832d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:11.074611 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.074507 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 21:24:11.074611 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.074516 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 21:24:11.074611 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.074519 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 21:24:11.074611 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.074537 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9ueslha3sv02m\"" Oct 30 21:24:11.074611 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.074566 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 21:24:11.074611 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.074580 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 21:24:11.074963 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.074660 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 21:24:11.074963 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.074722 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 21:24:11.074963 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.074740 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 21:24:11.074963 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.074746 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 21:24:11.074963 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.074759 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 21:24:11.130200 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.128613 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:24:11.130200 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.129584 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 21:24:11.130772 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.130749 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 21:24:11.174744 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.174701 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b287c83b-aa37-4063-9782-b62b202832d5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b287c83b-aa37-4063-9782-b62b202832d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:11.174958 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.174749 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b287c83b-aa37-4063-9782-b62b202832d5-config\") pod \"prometheus-k8s-0\" (UID: \"b287c83b-aa37-4063-9782-b62b202832d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:11.174958 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.174777 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b287c83b-aa37-4063-9782-b62b202832d5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b287c83b-aa37-4063-9782-b62b202832d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:11.174958 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.174806 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b287c83b-aa37-4063-9782-b62b202832d5-web-config\") pod \"prometheus-k8s-0\" (UID: \"b287c83b-aa37-4063-9782-b62b202832d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:11.174958 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.174832 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b287c83b-aa37-4063-9782-b62b202832d5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b287c83b-aa37-4063-9782-b62b202832d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:11.174958 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.174857 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b287c83b-aa37-4063-9782-b62b202832d5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b287c83b-aa37-4063-9782-b62b202832d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:11.174958 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.174882 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b287c83b-aa37-4063-9782-b62b202832d5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b287c83b-aa37-4063-9782-b62b202832d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:11.175275 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.174969 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b287c83b-aa37-4063-9782-b62b202832d5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b287c83b-aa37-4063-9782-b62b202832d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:11.175275 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.175009 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b287c83b-aa37-4063-9782-b62b202832d5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b287c83b-aa37-4063-9782-b62b202832d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:11.175275 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.175037 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b287c83b-aa37-4063-9782-b62b202832d5-config-out\") pod \"prometheus-k8s-0\" (UID: \"b287c83b-aa37-4063-9782-b62b202832d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:11.175275 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.175062 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b287c83b-aa37-4063-9782-b62b202832d5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b287c83b-aa37-4063-9782-b62b202832d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:11.175275 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.175087 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b287c83b-aa37-4063-9782-b62b202832d5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b287c83b-aa37-4063-9782-b62b202832d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:11.175275 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.175110 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b287c83b-aa37-4063-9782-b62b202832d5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b287c83b-aa37-4063-9782-b62b202832d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:11.175275 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.175150 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b287c83b-aa37-4063-9782-b62b202832d5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b287c83b-aa37-4063-9782-b62b202832d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:11.175275 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.175177 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b287c83b-aa37-4063-9782-b62b202832d5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b287c83b-aa37-4063-9782-b62b202832d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:11.175763 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.175675 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b287c83b-aa37-4063-9782-b62b202832d5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b287c83b-aa37-4063-9782-b62b202832d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:11.176186 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.175817 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b287c83b-aa37-4063-9782-b62b202832d5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b287c83b-aa37-4063-9782-b62b202832d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:11.176186 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.175872 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b287c83b-aa37-4063-9782-b62b202832d5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b287c83b-aa37-4063-9782-b62b202832d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:11.176186 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.175949 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b287c83b-aa37-4063-9782-b62b202832d5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b287c83b-aa37-4063-9782-b62b202832d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:11.176186 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.175986 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b287c83b-aa37-4063-9782-b62b202832d5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b287c83b-aa37-4063-9782-b62b202832d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:11.176186 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.176018 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b287c83b-aa37-4063-9782-b62b202832d5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b287c83b-aa37-4063-9782-b62b202832d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:11.176186 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.176046 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7cgl\" (UniqueName: \"kubernetes.io/projected/b287c83b-aa37-4063-9782-b62b202832d5-kube-api-access-c7cgl\") pod \"prometheus-k8s-0\" (UID: \"b287c83b-aa37-4063-9782-b62b202832d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:11.177175 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.176567 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b287c83b-aa37-4063-9782-b62b202832d5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b287c83b-aa37-4063-9782-b62b202832d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:11.177175 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.176578 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b287c83b-aa37-4063-9782-b62b202832d5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b287c83b-aa37-4063-9782-b62b202832d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:11.177175 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.176827 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b287c83b-aa37-4063-9782-b62b202832d5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b287c83b-aa37-4063-9782-b62b202832d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:11.177175 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.177133 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b287c83b-aa37-4063-9782-b62b202832d5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b287c83b-aa37-4063-9782-b62b202832d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:11.177403 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.177347 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b287c83b-aa37-4063-9782-b62b202832d5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b287c83b-aa37-4063-9782-b62b202832d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:11.177600 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.177573 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b287c83b-aa37-4063-9782-b62b202832d5-config-out\") pod \"prometheus-k8s-0\" (UID: \"b287c83b-aa37-4063-9782-b62b202832d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:11.177944 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.177923 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b287c83b-aa37-4063-9782-b62b202832d5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b287c83b-aa37-4063-9782-b62b202832d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:11.178006 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.177958 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b287c83b-aa37-4063-9782-b62b202832d5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b287c83b-aa37-4063-9782-b62b202832d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:11.179556 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.179539 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b287c83b-aa37-4063-9782-b62b202832d5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b287c83b-aa37-4063-9782-b62b202832d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:11.183741 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.183716 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b287c83b-aa37-4063-9782-b62b202832d5-config\") pod \"prometheus-k8s-0\" (UID: \"b287c83b-aa37-4063-9782-b62b202832d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:11.184126 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.184106 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b287c83b-aa37-4063-9782-b62b202832d5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b287c83b-aa37-4063-9782-b62b202832d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:11.184212 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.184192 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b287c83b-aa37-4063-9782-b62b202832d5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b287c83b-aa37-4063-9782-b62b202832d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:11.184255 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.184223 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b287c83b-aa37-4063-9782-b62b202832d5-web-config\") pod \"prometheus-k8s-0\" (UID: \"b287c83b-aa37-4063-9782-b62b202832d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:11.184255 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.184231 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b287c83b-aa37-4063-9782-b62b202832d5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b287c83b-aa37-4063-9782-b62b202832d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:11.184255 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.184226 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b287c83b-aa37-4063-9782-b62b202832d5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b287c83b-aa37-4063-9782-b62b202832d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:11.184540 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.184525 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b287c83b-aa37-4063-9782-b62b202832d5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b287c83b-aa37-4063-9782-b62b202832d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:11.184577 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.184546 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b287c83b-aa37-4063-9782-b62b202832d5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b287c83b-aa37-4063-9782-b62b202832d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:11.238465 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.238396 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7cgl\" (UniqueName: \"kubernetes.io/projected/b287c83b-aa37-4063-9782-b62b202832d5-kube-api-access-c7cgl\") pod \"prometheus-k8s-0\" (UID: \"b287c83b-aa37-4063-9782-b62b202832d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:11.425039 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.425011 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:11.877112 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:11.876881 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:24:11.879518 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:24:11.879489 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb287c83b_aa37_4063_9782_b62b202832d5.slice/crio-fd6d11200045d8214f7bb29f6dbc3cf5406902cf630758f69bd09a009c673357 WatchSource:0}: Error finding container fd6d11200045d8214f7bb29f6dbc3cf5406902cf630758f69bd09a009c673357: Status 404 returned error can't find the container with id fd6d11200045d8214f7bb29f6dbc3cf5406902cf630758f69bd09a009c673357 Oct 30 21:24:12.414225 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:12.414192 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bce4cfcf-5926-433e-a0dd-238dfbb6ef1c" path="/var/lib/kubelet/pods/bce4cfcf-5926-433e-a0dd-238dfbb6ef1c/volumes" Oct 30 21:24:12.730926 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:12.730827 5748 generic.go:356] "Generic (PLEG): container finished" podID="b287c83b-aa37-4063-9782-b62b202832d5" containerID="a041288b1379a52a563c0f691ee897b1790af5c0f87b1a4e9c2b611eba70e15f" exitCode=0 Oct 30 21:24:12.731215 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:12.730932 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b287c83b-aa37-4063-9782-b62b202832d5","Type":"ContainerDied","Data":"a041288b1379a52a563c0f691ee897b1790af5c0f87b1a4e9c2b611eba70e15f"} Oct 30 21:24:12.731215 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:12.731021 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b287c83b-aa37-4063-9782-b62b202832d5","Type":"ContainerStarted","Data":"fd6d11200045d8214f7bb29f6dbc3cf5406902cf630758f69bd09a009c673357"} Oct 30 21:24:13.738078 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:13.738043 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b287c83b-aa37-4063-9782-b62b202832d5","Type":"ContainerStarted","Data":"8bd7285c781ae0e944fc94ba1af94bf1f45c2715c02129cfb9f83437e4f6ebf6"} Oct 30 21:24:13.738652 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:13.738133 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b287c83b-aa37-4063-9782-b62b202832d5","Type":"ContainerStarted","Data":"04af0916176d2ca30e1e2732d7f9f4d631596f15939982d157416504d0603064"} Oct 30 21:24:13.738652 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:13.738166 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b287c83b-aa37-4063-9782-b62b202832d5","Type":"ContainerStarted","Data":"54ea3eda867e073aa493e2c7fd7e846a5e8e280e299bbb5236c7717d9f219789"} Oct 30 21:24:13.738652 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:13.738190 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b287c83b-aa37-4063-9782-b62b202832d5","Type":"ContainerStarted","Data":"68c4e75b795bd770fcba1818b3a112b8e6b4137c255cc3dad73c102a7ff6a13e"} Oct 30 21:24:13.738652 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:13.738213 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b287c83b-aa37-4063-9782-b62b202832d5","Type":"ContainerStarted","Data":"d9144dd5848f46547db062fa2c5c8687036912a0fd12f76e945e6e224974516e"} Oct 30 21:24:13.738652 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:13.738236 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b287c83b-aa37-4063-9782-b62b202832d5","Type":"ContainerStarted","Data":"555a8931f01b1d48b4594809e238d6c9928035f381280bbf97dc71fe35d70303"} Oct 30 21:24:13.872957 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:13.872875 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.855246089 podStartE2EDuration="3.872861019s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.61493ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:13.872447131 +0000 UTC m=+37.614968066" watchObservedRunningTime="2026-10-30 21:24:13.872861019 +0000 UTC m=+37.615381996" Oct 30 21:24:14.935698 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:14.935666 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5dcb975b97-crpzf" Oct 30 21:24:14.939543 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:14.939514 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5dcb975b97-crpzf" Oct 30 21:24:16.425164 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:16.425113 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:20.936292 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:20.936228 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7fbd54bb44-d7zl5" podUID="30b586ee-8d8a-4e95-b9a3-64f3b09c77e6" containerName="console" containerID="cri-o://af535649f2985c0b00d326fada3f0ffa17d782627e119712f895c56b22fad00e" gracePeriod=15 Oct 30 21:24:21.231498 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:21.231469 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fbd54bb44-d7zl5" Oct 30 21:24:21.378686 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:21.378653 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-kube-api-access-4nk9v\" (UniqueName: \"kubernetes.io/projected/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-kube-api-access-4nk9v\") pod \"30b586ee-8d8a-4e95-b9a3-64f3b09c77e6\" (UID: \"30b586ee-8d8a-4e95-b9a3-64f3b09c77e6\") " Oct 30 21:24:21.378892 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:21.378704 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-service-ca\" (UniqueName: \"kubernetes.io/configmap/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-service-ca\") pod \"30b586ee-8d8a-4e95-b9a3-64f3b09c77e6\" (UID: \"30b586ee-8d8a-4e95-b9a3-64f3b09c77e6\") " Oct 30 21:24:21.378892 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:21.378773 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-console-serving-cert\") pod \"30b586ee-8d8a-4e95-b9a3-64f3b09c77e6\" (UID: \"30b586ee-8d8a-4e95-b9a3-64f3b09c77e6\") " Oct 30 21:24:21.378892 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:21.378818 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-console-config\" (UniqueName: \"kubernetes.io/configmap/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-console-config\") pod \"30b586ee-8d8a-4e95-b9a3-64f3b09c77e6\" (UID: \"30b586ee-8d8a-4e95-b9a3-64f3b09c77e6\") " Oct 30 21:24:21.378892 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:21.378848 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-oauth-serving-cert\") pod \"30b586ee-8d8a-4e95-b9a3-64f3b09c77e6\" (UID: \"30b586ee-8d8a-4e95-b9a3-64f3b09c77e6\") " Oct 30 21:24:21.379143 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:21.378926 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-tmp\" (UniqueName: \"kubernetes.io/empty-dir/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-tmp\") pod \"30b586ee-8d8a-4e95-b9a3-64f3b09c77e6\" (UID: \"30b586ee-8d8a-4e95-b9a3-64f3b09c77e6\") " Oct 30 21:24:21.379143 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:21.379002 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-console-oauth-config\") pod \"30b586ee-8d8a-4e95-b9a3-64f3b09c77e6\" (UID: \"30b586ee-8d8a-4e95-b9a3-64f3b09c77e6\") " Oct 30 21:24:21.379330 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:21.379218 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-service-ca" pod "30b586ee-8d8a-4e95-b9a3-64f3b09c77e6" (UID: "30b586ee-8d8a-4e95-b9a3-64f3b09c77e6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:24:21.379330 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:21.379220 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-console-config" pod "30b586ee-8d8a-4e95-b9a3-64f3b09c77e6" (UID: "30b586ee-8d8a-4e95-b9a3-64f3b09c77e6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:24:21.379467 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:21.379302 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-tmp" pod "30b586ee-8d8a-4e95-b9a3-64f3b09c77e6" (UID: "30b586ee-8d8a-4e95-b9a3-64f3b09c77e6"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:24:21.379467 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:21.379339 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-oauth-serving-cert" pod "30b586ee-8d8a-4e95-b9a3-64f3b09c77e6" (UID: "30b586ee-8d8a-4e95-b9a3-64f3b09c77e6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:24:21.381166 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:21.381140 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-kube-api-access-4nk9v" pod "30b586ee-8d8a-4e95-b9a3-64f3b09c77e6" (UID: "30b586ee-8d8a-4e95-b9a3-64f3b09c77e6"). InnerVolumeSpecName "kube-api-access-4nk9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:24:21.381166 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:21.381157 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-console-serving-cert" pod "30b586ee-8d8a-4e95-b9a3-64f3b09c77e6" (UID: "30b586ee-8d8a-4e95-b9a3-64f3b09c77e6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:24:21.381282 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:21.381221 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-console-oauth-config" pod "30b586ee-8d8a-4e95-b9a3-64f3b09c77e6" (UID: "30b586ee-8d8a-4e95-b9a3-64f3b09c77e6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:24:21.480091 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:21.480007 5748 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-console-config\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:21.480091 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:21.480039 5748 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-oauth-serving-cert\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:21.480091 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:21.480050 5748 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-tmp\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:21.480091 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:21.480062 5748 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-console-oauth-config\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:21.480091 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:21.480070 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nk9v\" (UniqueName: \"kubernetes.io/projected/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-kube-api-access-4nk9v\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:21.480091 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:21.480080 5748 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-service-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:21.480091 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:21.480094 5748 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6-console-serving-cert\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:21.774152 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:21.774068 5748 generic.go:356] "Generic (PLEG): container finished" podID="30b586ee-8d8a-4e95-b9a3-64f3b09c77e6" containerID="af535649f2985c0b00d326fada3f0ffa17d782627e119712f895c56b22fad00e" exitCode=0 Oct 30 21:24:21.774152 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:21.774120 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fbd54bb44-d7zl5" Oct 30 21:24:21.774152 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:21.774130 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fbd54bb44-d7zl5" event={"ID":"30b586ee-8d8a-4e95-b9a3-64f3b09c77e6","Type":"ContainerDied","Data":"af535649f2985c0b00d326fada3f0ffa17d782627e119712f895c56b22fad00e"} Oct 30 21:24:21.774372 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:21.774174 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fbd54bb44-d7zl5" event={"ID":"30b586ee-8d8a-4e95-b9a3-64f3b09c77e6","Type":"ContainerDied","Data":"90b5c7445469629806ce6590748554079f89b212ed99dfc9f33c2f39aea5951c"} Oct 30 21:24:21.774372 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:21.774198 5748 scope.go:122] "RemoveContainer" containerID="af535649f2985c0b00d326fada3f0ffa17d782627e119712f895c56b22fad00e" Oct 30 21:24:21.783569 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:21.783544 5748 scope.go:122] "RemoveContainer" containerID="af535649f2985c0b00d326fada3f0ffa17d782627e119712f895c56b22fad00e" Oct 30 21:24:21.785015 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:24:21.784990 5748 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af535649f2985c0b00d326fada3f0ffa17d782627e119712f895c56b22fad00e\": container with ID starting with af535649f2985c0b00d326fada3f0ffa17d782627e119712f895c56b22fad00e not found: ID does not exist" containerID="af535649f2985c0b00d326fada3f0ffa17d782627e119712f895c56b22fad00e" Oct 30 21:24:21.785134 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:21.785022 5748 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af535649f2985c0b00d326fada3f0ffa17d782627e119712f895c56b22fad00e"} err="failed to get container status \"af535649f2985c0b00d326fada3f0ffa17d782627e119712f895c56b22fad00e\": rpc error: code = NotFound desc = could not find container \"af535649f2985c0b00d326fada3f0ffa17d782627e119712f895c56b22fad00e\": container with ID starting with af535649f2985c0b00d326fada3f0ffa17d782627e119712f895c56b22fad00e not found: ID does not exist" Oct 30 21:24:21.940404 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:21.940368 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7fbd54bb44-d7zl5"] Oct 30 21:24:21.990867 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:21.990835 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7fbd54bb44-d7zl5"] Oct 30 21:24:22.414461 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:22.414424 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30b586ee-8d8a-4e95-b9a3-64f3b09c77e6" path="/var/lib/kubelet/pods/30b586ee-8d8a-4e95-b9a3-64f3b09c77e6/volumes" Oct 30 21:24:30.144350 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:24:30.144305 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:24:57.845977 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:57.845936 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:57.861829 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:57.861800 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:24:57.923249 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:24:57.923223 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:25:00.144043 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:25:00.144010 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:25:30.143356 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:25:30.143318 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:26:00.144617 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:26:00.144578 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:26:30.144436 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:26:30.144390 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:27:00.144753 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:27:00.144714 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:27:18.280622 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:18.280584 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-568bcd9cdd-sl9dg"] Oct 30 21:27:18.281060 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:18.280947 5748 cpu_manager.go:414] "containerMap: removing container" podUID="30b586ee-8d8a-4e95-b9a3-64f3b09c77e6" containerName="console" Oct 30 21:27:18.281060 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:18.280959 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30b586ee-8d8a-4e95-b9a3-64f3b09c77e6" containerName="console" Oct 30 21:27:18.281060 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:18.281045 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="30b586ee-8d8a-4e95-b9a3-64f3b09c77e6" containerName="console" Oct 30 21:27:18.282952 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:18.282935 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-568bcd9cdd-sl9dg" Oct 30 21:27:18.305672 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:18.305633 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4hzd\" (UniqueName: \"kubernetes.io/projected/b678b0b8-bf45-49ea-9f34-0751719778e8-kube-api-access-g4hzd\") pod \"opendatahub-operator-controller-manager-568bcd9cdd-sl9dg\" (UID: \"b678b0b8-bf45-49ea-9f34-0751719778e8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-568bcd9cdd-sl9dg" Oct 30 21:27:18.305815 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:18.305690 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b678b0b8-bf45-49ea-9f34-0751719778e8-webhook-cert\") pod \"opendatahub-operator-controller-manager-568bcd9cdd-sl9dg\" (UID: \"b678b0b8-bf45-49ea-9f34-0751719778e8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-568bcd9cdd-sl9dg" Oct 30 21:27:18.305815 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:18.305773 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b678b0b8-bf45-49ea-9f34-0751719778e8-tmp\") pod \"opendatahub-operator-controller-manager-568bcd9cdd-sl9dg\" (UID: \"b678b0b8-bf45-49ea-9f34-0751719778e8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-568bcd9cdd-sl9dg" Oct 30 21:27:18.305889 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:18.305826 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b678b0b8-bf45-49ea-9f34-0751719778e8-metrics-cert\") pod \"opendatahub-operator-controller-manager-568bcd9cdd-sl9dg\" (UID: \"b678b0b8-bf45-49ea-9f34-0751719778e8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-568bcd9cdd-sl9dg" Oct 30 21:27:18.305889 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:18.305877 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b678b0b8-bf45-49ea-9f34-0751719778e8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-568bcd9cdd-sl9dg\" (UID: \"b678b0b8-bf45-49ea-9f34-0751719778e8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-568bcd9cdd-sl9dg" Oct 30 21:27:18.305980 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:18.305952 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b678b0b8-bf45-49ea-9f34-0751719778e8-manifests\") pod \"opendatahub-operator-controller-manager-568bcd9cdd-sl9dg\" (UID: \"b678b0b8-bf45-49ea-9f34-0751719778e8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-568bcd9cdd-sl9dg" Oct 30 21:27:18.388550 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:18.388511 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-568bcd9cdd-sl9dg"] Oct 30 21:27:18.406541 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:18.406487 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b678b0b8-bf45-49ea-9f34-0751719778e8-webhook-cert\") pod \"opendatahub-operator-controller-manager-568bcd9cdd-sl9dg\" (UID: \"b678b0b8-bf45-49ea-9f34-0751719778e8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-568bcd9cdd-sl9dg" Oct 30 21:27:18.406744 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:18.406550 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b678b0b8-bf45-49ea-9f34-0751719778e8-tmp\") pod \"opendatahub-operator-controller-manager-568bcd9cdd-sl9dg\" (UID: \"b678b0b8-bf45-49ea-9f34-0751719778e8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-568bcd9cdd-sl9dg" Oct 30 21:27:18.406744 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:18.406605 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b678b0b8-bf45-49ea-9f34-0751719778e8-metrics-cert\") pod \"opendatahub-operator-controller-manager-568bcd9cdd-sl9dg\" (UID: \"b678b0b8-bf45-49ea-9f34-0751719778e8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-568bcd9cdd-sl9dg" Oct 30 21:27:18.406744 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:18.406686 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b678b0b8-bf45-49ea-9f34-0751719778e8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-568bcd9cdd-sl9dg\" (UID: \"b678b0b8-bf45-49ea-9f34-0751719778e8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-568bcd9cdd-sl9dg" Oct 30 21:27:18.406744 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:18.406707 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b678b0b8-bf45-49ea-9f34-0751719778e8-manifests\") pod \"opendatahub-operator-controller-manager-568bcd9cdd-sl9dg\" (UID: \"b678b0b8-bf45-49ea-9f34-0751719778e8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-568bcd9cdd-sl9dg" Oct 30 21:27:18.406954 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:18.406759 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4hzd\" (UniqueName: \"kubernetes.io/projected/b678b0b8-bf45-49ea-9f34-0751719778e8-kube-api-access-g4hzd\") pod \"opendatahub-operator-controller-manager-568bcd9cdd-sl9dg\" (UID: \"b678b0b8-bf45-49ea-9f34-0751719778e8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-568bcd9cdd-sl9dg" Oct 30 21:27:18.407149 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:18.407124 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b678b0b8-bf45-49ea-9f34-0751719778e8-manifests\") pod \"opendatahub-operator-controller-manager-568bcd9cdd-sl9dg\" (UID: \"b678b0b8-bf45-49ea-9f34-0751719778e8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-568bcd9cdd-sl9dg" Oct 30 21:27:18.408068 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:18.408036 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 21:27:18.408217 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:18.408091 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 21:27:18.408217 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:18.408198 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 21:27:18.408217 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:18.408200 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 21:27:18.408380 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:18.408275 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 21:27:18.408380 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:18.408350 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-9g9fn\"" Oct 30 21:27:18.408933 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:18.408892 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b678b0b8-bf45-49ea-9f34-0751719778e8-tmp\") pod \"opendatahub-operator-controller-manager-568bcd9cdd-sl9dg\" (UID: \"b678b0b8-bf45-49ea-9f34-0751719778e8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-568bcd9cdd-sl9dg" Oct 30 21:27:18.420287 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:18.420245 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b678b0b8-bf45-49ea-9f34-0751719778e8-metrics-cert\") pod \"opendatahub-operator-controller-manager-568bcd9cdd-sl9dg\" (UID: \"b678b0b8-bf45-49ea-9f34-0751719778e8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-568bcd9cdd-sl9dg" Oct 30 21:27:18.421334 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:18.421308 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b678b0b8-bf45-49ea-9f34-0751719778e8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-568bcd9cdd-sl9dg\" (UID: \"b678b0b8-bf45-49ea-9f34-0751719778e8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-568bcd9cdd-sl9dg" Oct 30 21:27:18.422141 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:18.422108 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b678b0b8-bf45-49ea-9f34-0751719778e8-webhook-cert\") pod \"opendatahub-operator-controller-manager-568bcd9cdd-sl9dg\" (UID: \"b678b0b8-bf45-49ea-9f34-0751719778e8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-568bcd9cdd-sl9dg" Oct 30 21:27:18.473611 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:18.473570 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4hzd\" (UniqueName: \"kubernetes.io/projected/b678b0b8-bf45-49ea-9f34-0751719778e8-kube-api-access-g4hzd\") pod \"opendatahub-operator-controller-manager-568bcd9cdd-sl9dg\" (UID: \"b678b0b8-bf45-49ea-9f34-0751719778e8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-568bcd9cdd-sl9dg" Oct 30 21:27:18.596619 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:18.596530 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-568bcd9cdd-sl9dg" Oct 30 21:27:19.104998 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:19.104814 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-568bcd9cdd-sl9dg"] Oct 30 21:27:19.107858 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:27:19.107828 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb678b0b8_bf45_49ea_9f34_0751719778e8.slice/crio-c9b18e585e92ec03999f02b7b2628a50af3cc826766c871ce7575fba355ab1d6 WatchSource:0}: Error finding container c9b18e585e92ec03999f02b7b2628a50af3cc826766c871ce7575fba355ab1d6: Status 404 returned error can't find the container with id c9b18e585e92ec03999f02b7b2628a50af3cc826766c871ce7575fba355ab1d6 Oct 30 21:27:19.441690 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:19.441650 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-568bcd9cdd-sl9dg" event={"ID":"b678b0b8-bf45-49ea-9f34-0751719778e8","Type":"ContainerStarted","Data":"c9b18e585e92ec03999f02b7b2628a50af3cc826766c871ce7575fba355ab1d6"} Oct 30 21:27:22.856530 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:22.856441 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-bh2l5"] Oct 30 21:27:22.860559 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:22.860536 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bh2l5" Oct 30 21:27:22.920248 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:22.920219 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-bh2l5"] Oct 30 21:27:22.948033 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:22.947999 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88fr7\" (UniqueName: \"kubernetes.io/projected/62a4e725-cffd-4326-b072-506423d26fc5-kube-api-access-88fr7\") pod \"cert-manager-webhook-5dbbbf75cc-bh2l5\" (UID: \"62a4e725-cffd-4326-b072-506423d26fc5\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bh2l5" Oct 30 21:27:22.948201 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:22.948041 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/62a4e725-cffd-4326-b072-506423d26fc5-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-bh2l5\" (UID: \"62a4e725-cffd-4326-b072-506423d26fc5\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bh2l5" Oct 30 21:27:22.966235 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:22.966204 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 21:27:22.966235 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:22.966214 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 21:27:22.966235 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:22.966237 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-tthsr\"" Oct 30 21:27:23.048715 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:23.048682 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88fr7\" (UniqueName: \"kubernetes.io/projected/62a4e725-cffd-4326-b072-506423d26fc5-kube-api-access-88fr7\") pod \"cert-manager-webhook-5dbbbf75cc-bh2l5\" (UID: \"62a4e725-cffd-4326-b072-506423d26fc5\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bh2l5" Oct 30 21:27:23.048715 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:23.048722 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/62a4e725-cffd-4326-b072-506423d26fc5-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-bh2l5\" (UID: \"62a4e725-cffd-4326-b072-506423d26fc5\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bh2l5" Oct 30 21:27:23.114487 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:23.114405 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88fr7\" (UniqueName: \"kubernetes.io/projected/62a4e725-cffd-4326-b072-506423d26fc5-kube-api-access-88fr7\") pod \"cert-manager-webhook-5dbbbf75cc-bh2l5\" (UID: \"62a4e725-cffd-4326-b072-506423d26fc5\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bh2l5" Oct 30 21:27:23.118171 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:23.118147 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/62a4e725-cffd-4326-b072-506423d26fc5-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-bh2l5\" (UID: \"62a4e725-cffd-4326-b072-506423d26fc5\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bh2l5" Oct 30 21:27:23.182714 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:23.182673 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bh2l5" Oct 30 21:27:23.463914 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:23.463862 5748 generic.go:356] "Generic (PLEG): container finished" podID="b678b0b8-bf45-49ea-9f34-0751719778e8" containerID="47d53b337dd7ff706f30c50c84f00a3200a64afd209d84dc13ddbf9367dc4227" exitCode=0 Oct 30 21:27:23.464100 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:23.463933 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-568bcd9cdd-sl9dg" event={"ID":"b678b0b8-bf45-49ea-9f34-0751719778e8","Type":"ContainerDied","Data":"47d53b337dd7ff706f30c50c84f00a3200a64afd209d84dc13ddbf9367dc4227"} Oct 30 21:27:23.558990 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:27:23.558958 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod62a4e725_cffd_4326_b072_506423d26fc5.slice/crio-de4e0a1ae0dfde012c8278df04a26c2ac03f8494766e0695fa33307e7cef480a WatchSource:0}: Error finding container de4e0a1ae0dfde012c8278df04a26c2ac03f8494766e0695fa33307e7cef480a: Status 404 returned error can't find the container with id de4e0a1ae0dfde012c8278df04a26c2ac03f8494766e0695fa33307e7cef480a Oct 30 21:27:23.569241 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:23.569218 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-bh2l5"] Oct 30 21:27:24.470252 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:24.470215 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-568bcd9cdd-sl9dg" event={"ID":"b678b0b8-bf45-49ea-9f34-0751719778e8","Type":"ContainerStarted","Data":"ec6d1a157e4cb747b67893fd9969f1431c5dc089bf999c45df4bc02a4c3f31c0"} Oct 30 21:27:24.470853 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:24.470305 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-568bcd9cdd-sl9dg" Oct 30 21:27:24.471595 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:24.471573 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bh2l5" event={"ID":"62a4e725-cffd-4326-b072-506423d26fc5","Type":"ContainerStarted","Data":"de4e0a1ae0dfde012c8278df04a26c2ac03f8494766e0695fa33307e7cef480a"} Oct 30 21:27:24.743408 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:24.743349 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-568bcd9cdd-sl9dg" podStartSLOduration=2.205537986 podStartE2EDuration="6.74333005s" totalImagesPullingTime="4.128902148s" totalInitContainerRuntime="408.889916ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:24.69241508 +0000 UTC m=+228.434936017" watchObservedRunningTime="2026-10-30 21:27:24.74333005 +0000 UTC m=+228.485850985" Oct 30 21:27:25.040213 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:25.040135 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-pwwtb"] Oct 30 21:27:25.042284 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:25.042267 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-pwwtb" Oct 30 21:27:25.067610 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:25.067571 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/46007704-5f70-457b-978b-6650fb213660-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-pwwtb\" (UID: \"46007704-5f70-457b-978b-6650fb213660\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-pwwtb" Oct 30 21:27:25.067816 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:25.067788 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf66k\" (UniqueName: \"kubernetes.io/projected/46007704-5f70-457b-978b-6650fb213660-kube-api-access-pf66k\") pod \"cert-manager-cainjector-5c57bbd4fc-pwwtb\" (UID: \"46007704-5f70-457b-978b-6650fb213660\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-pwwtb" Oct 30 21:27:25.104436 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:25.104395 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-zl5s6\"" Oct 30 21:27:25.138514 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:25.138485 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-pwwtb"] Oct 30 21:27:25.168497 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:25.168459 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf66k\" (UniqueName: \"kubernetes.io/projected/46007704-5f70-457b-978b-6650fb213660-kube-api-access-pf66k\") pod \"cert-manager-cainjector-5c57bbd4fc-pwwtb\" (UID: \"46007704-5f70-457b-978b-6650fb213660\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-pwwtb" Oct 30 21:27:25.168700 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:25.168540 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/46007704-5f70-457b-978b-6650fb213660-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-pwwtb\" (UID: \"46007704-5f70-457b-978b-6650fb213660\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-pwwtb" Oct 30 21:27:25.241081 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:25.241049 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf66k\" (UniqueName: \"kubernetes.io/projected/46007704-5f70-457b-978b-6650fb213660-kube-api-access-pf66k\") pod \"cert-manager-cainjector-5c57bbd4fc-pwwtb\" (UID: \"46007704-5f70-457b-978b-6650fb213660\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-pwwtb" Oct 30 21:27:25.248237 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:25.248208 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/46007704-5f70-457b-978b-6650fb213660-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-pwwtb\" (UID: \"46007704-5f70-457b-978b-6650fb213660\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-pwwtb" Oct 30 21:27:25.358594 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:25.358497 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-pwwtb" Oct 30 21:27:25.739440 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:25.739404 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-pwwtb"] Oct 30 21:27:25.744410 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:27:25.744372 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod46007704_5f70_457b_978b_6650fb213660.slice/crio-f7935681dd0f1756400d18ddf78b1837e53034e723e311f7cea4b6c5db27497b WatchSource:0}: Error finding container f7935681dd0f1756400d18ddf78b1837e53034e723e311f7cea4b6c5db27497b: Status 404 returned error can't find the container with id f7935681dd0f1756400d18ddf78b1837e53034e723e311f7cea4b6c5db27497b Oct 30 21:27:26.482968 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:26.482933 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-pwwtb" event={"ID":"46007704-5f70-457b-978b-6650fb213660","Type":"ContainerStarted","Data":"f7935681dd0f1756400d18ddf78b1837e53034e723e311f7cea4b6c5db27497b"} Oct 30 21:27:28.494128 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:28.494093 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-pwwtb" event={"ID":"46007704-5f70-457b-978b-6650fb213660","Type":"ContainerStarted","Data":"ae34d843007b1262b6b09af8ef0535fcd973178060cdedddc712bb615d234cdd"} Oct 30 21:27:28.495949 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:28.495924 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bh2l5" event={"ID":"62a4e725-cffd-4326-b072-506423d26fc5","Type":"ContainerStarted","Data":"292870d8b2a1051cb198ea675f2b1312c939b51008276e598e95c8f9976c3d48"} Oct 30 21:27:28.496057 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:28.495975 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bh2l5" Oct 30 21:27:28.613175 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:28.613122 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-pwwtb" podStartSLOduration=1.940640852 podStartE2EDuration="4.613106888s" totalImagesPullingTime="2.672466036s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:28.612673038 +0000 UTC m=+232.355193974" watchObservedRunningTime="2026-10-30 21:27:28.613106888 +0000 UTC m=+232.355627819" Oct 30 21:27:28.732022 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:28.731919 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bh2l5" podStartSLOduration=2.555014071 podStartE2EDuration="6.731880177s" totalImagesPullingTime="4.176866106s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:28.730946852 +0000 UTC m=+232.473467795" watchObservedRunningTime="2026-10-30 21:27:28.731880177 +0000 UTC m=+232.474401112" Oct 30 21:27:30.143415 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:27:30.143369 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:27:34.052038 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:34.052000 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-tc6wb"] Oct 30 21:27:34.055704 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:34.055686 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-tc6wb" Oct 30 21:27:34.148201 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:34.148164 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3e5be9e5-9450-4c16-831e-a81c23014be4-bound-sa-token\") pod \"cert-manager-597b95cb84-tc6wb\" (UID: \"3e5be9e5-9450-4c16-831e-a81c23014be4\") " pod="cert-manager/cert-manager-597b95cb84-tc6wb" Oct 30 21:27:34.148378 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:34.148316 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4d6g\" (UniqueName: \"kubernetes.io/projected/3e5be9e5-9450-4c16-831e-a81c23014be4-kube-api-access-p4d6g\") pod \"cert-manager-597b95cb84-tc6wb\" (UID: \"3e5be9e5-9450-4c16-831e-a81c23014be4\") " pod="cert-manager/cert-manager-597b95cb84-tc6wb" Oct 30 21:27:34.159150 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:34.159120 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-sf24f\"" Oct 30 21:27:34.160177 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:34.160154 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-tc6wb"] Oct 30 21:27:34.249061 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:34.249027 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4d6g\" (UniqueName: \"kubernetes.io/projected/3e5be9e5-9450-4c16-831e-a81c23014be4-kube-api-access-p4d6g\") pod \"cert-manager-597b95cb84-tc6wb\" (UID: \"3e5be9e5-9450-4c16-831e-a81c23014be4\") " pod="cert-manager/cert-manager-597b95cb84-tc6wb" Oct 30 21:27:34.249236 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:34.249076 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3e5be9e5-9450-4c16-831e-a81c23014be4-bound-sa-token\") pod \"cert-manager-597b95cb84-tc6wb\" (UID: \"3e5be9e5-9450-4c16-831e-a81c23014be4\") " pod="cert-manager/cert-manager-597b95cb84-tc6wb" Oct 30 21:27:34.310744 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:34.310671 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4d6g\" (UniqueName: \"kubernetes.io/projected/3e5be9e5-9450-4c16-831e-a81c23014be4-kube-api-access-p4d6g\") pod \"cert-manager-597b95cb84-tc6wb\" (UID: \"3e5be9e5-9450-4c16-831e-a81c23014be4\") " pod="cert-manager/cert-manager-597b95cb84-tc6wb" Oct 30 21:27:34.310854 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:34.310817 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3e5be9e5-9450-4c16-831e-a81c23014be4-bound-sa-token\") pod \"cert-manager-597b95cb84-tc6wb\" (UID: \"3e5be9e5-9450-4c16-831e-a81c23014be4\") " pod="cert-manager/cert-manager-597b95cb84-tc6wb" Oct 30 21:27:34.367926 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:34.367861 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-tc6wb" Oct 30 21:27:34.502230 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:34.502199 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bh2l5" Oct 30 21:27:34.740928 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:34.740885 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-tc6wb"] Oct 30 21:27:34.744237 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:27:34.744198 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3e5be9e5_9450_4c16_831e_a81c23014be4.slice/crio-59e4039b303b5d678fea95d61e2056396e67f4761a9194b6a6739ec7bf60e9ba WatchSource:0}: Error finding container 59e4039b303b5d678fea95d61e2056396e67f4761a9194b6a6739ec7bf60e9ba: Status 404 returned error can't find the container with id 59e4039b303b5d678fea95d61e2056396e67f4761a9194b6a6739ec7bf60e9ba Oct 30 21:27:35.479510 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:35.479477 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-568bcd9cdd-sl9dg" Oct 30 21:27:35.525062 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:35.525028 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-tc6wb" event={"ID":"3e5be9e5-9450-4c16-831e-a81c23014be4","Type":"ContainerStarted","Data":"af8509ccdd0ad940ee4fb05320e168dd996434c56fe95c8631651aafd307006c"} Oct 30 21:27:35.525062 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:35.525068 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-tc6wb" event={"ID":"3e5be9e5-9450-4c16-831e-a81c23014be4","Type":"ContainerStarted","Data":"59e4039b303b5d678fea95d61e2056396e67f4761a9194b6a6739ec7bf60e9ba"} Oct 30 21:27:35.847453 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:35.847352 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-tc6wb" podStartSLOduration=2.8473361759999998 podStartE2EDuration="2.847336176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:35.846177528 +0000 UTC m=+239.588698464" watchObservedRunningTime="2026-10-30 21:27:35.847336176 +0000 UTC m=+239.589857112" Oct 30 21:27:55.586570 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:55.586534 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-fm72c"] Oct 30 21:27:55.593725 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:55.593698 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fm72c" Oct 30 21:27:55.647401 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:55.647346 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 21:27:55.648442 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:55.648417 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-fm72c"] Oct 30 21:27:55.649887 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:55.649866 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Oct 30 21:27:55.697743 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:55.697706 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Oct 30 21:27:55.697963 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:55.697774 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-dpbcr\"" Oct 30 21:27:55.746046 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:55.746009 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8b9b71d4-264b-4666-bfe3-ff0dcffafc85-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-fm72c\" (UID: \"8b9b71d4-264b-4666-bfe3-ff0dcffafc85\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fm72c" Oct 30 21:27:55.746228 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:55.746062 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8b9b71d4-264b-4666-bfe3-ff0dcffafc85-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-fm72c\" (UID: \"8b9b71d4-264b-4666-bfe3-ff0dcffafc85\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fm72c" Oct 30 21:27:55.746228 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:55.746135 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlvfp\" (UniqueName: \"kubernetes.io/projected/8b9b71d4-264b-4666-bfe3-ff0dcffafc85-kube-api-access-xlvfp\") pod \"kuadrant-console-plugin-66c4fb575d-fm72c\" (UID: \"8b9b71d4-264b-4666-bfe3-ff0dcffafc85\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fm72c" Oct 30 21:27:55.747241 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:55.747224 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 21:27:55.847643 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:55.847554 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8b9b71d4-264b-4666-bfe3-ff0dcffafc85-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-fm72c\" (UID: \"8b9b71d4-264b-4666-bfe3-ff0dcffafc85\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fm72c" Oct 30 21:27:55.847643 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:55.847600 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlvfp\" (UniqueName: \"kubernetes.io/projected/8b9b71d4-264b-4666-bfe3-ff0dcffafc85-kube-api-access-xlvfp\") pod \"kuadrant-console-plugin-66c4fb575d-fm72c\" (UID: \"8b9b71d4-264b-4666-bfe3-ff0dcffafc85\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fm72c" Oct 30 21:27:55.847843 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:55.847728 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8b9b71d4-264b-4666-bfe3-ff0dcffafc85-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-fm72c\" (UID: \"8b9b71d4-264b-4666-bfe3-ff0dcffafc85\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fm72c" Oct 30 21:27:55.848425 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:55.848406 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8b9b71d4-264b-4666-bfe3-ff0dcffafc85-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-fm72c\" (UID: \"8b9b71d4-264b-4666-bfe3-ff0dcffafc85\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fm72c" Oct 30 21:27:55.850117 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:55.850099 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8b9b71d4-264b-4666-bfe3-ff0dcffafc85-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-fm72c\" (UID: \"8b9b71d4-264b-4666-bfe3-ff0dcffafc85\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fm72c" Oct 30 21:27:55.911360 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:55.911324 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlvfp\" (UniqueName: \"kubernetes.io/projected/8b9b71d4-264b-4666-bfe3-ff0dcffafc85-kube-api-access-xlvfp\") pod \"kuadrant-console-plugin-66c4fb575d-fm72c\" (UID: \"8b9b71d4-264b-4666-bfe3-ff0dcffafc85\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fm72c" Oct 30 21:27:56.204379 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:56.204335 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fm72c" Oct 30 21:27:56.575706 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:56.575629 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-fm72c"] Oct 30 21:27:56.578257 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:27:56.578225 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8b9b71d4_264b_4666_bfe3_ff0dcffafc85.slice/crio-43349b221585d324700ff09c1adb9cd46c958718513cc29eb1b12662f4661b62 WatchSource:0}: Error finding container 43349b221585d324700ff09c1adb9cd46c958718513cc29eb1b12662f4661b62: Status 404 returned error can't find the container with id 43349b221585d324700ff09c1adb9cd46c958718513cc29eb1b12662f4661b62 Oct 30 21:27:56.613415 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:56.613377 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fm72c" event={"ID":"8b9b71d4-264b-4666-bfe3-ff0dcffafc85","Type":"ContainerStarted","Data":"43349b221585d324700ff09c1adb9cd46c958718513cc29eb1b12662f4661b62"} Oct 30 21:27:59.360129 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:59.360087 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6bcd447999-f4qch"] Oct 30 21:27:59.362600 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:59.362573 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6bcd447999-f4qch" Oct 30 21:27:59.463028 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:59.462995 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 21:27:59.463028 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:59.462995 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 21:27:59.468161 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:59.468083 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6bcd447999-f4qch"] Oct 30 21:27:59.481792 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:59.481746 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e692bffa-b34f-4ce9-9db8-6667ed013afb-tmp\") pod \"kube-auth-proxy-6bcd447999-f4qch\" (UID: \"e692bffa-b34f-4ce9-9db8-6667ed013afb\") " pod="openshift-ingress/kube-auth-proxy-6bcd447999-f4qch" Oct 30 21:27:59.482006 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:59.481845 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxz6j\" (UniqueName: \"kubernetes.io/projected/e692bffa-b34f-4ce9-9db8-6667ed013afb-kube-api-access-dxz6j\") pod \"kube-auth-proxy-6bcd447999-f4qch\" (UID: \"e692bffa-b34f-4ce9-9db8-6667ed013afb\") " pod="openshift-ingress/kube-auth-proxy-6bcd447999-f4qch" Oct 30 21:27:59.482006 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:59.481887 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e692bffa-b34f-4ce9-9db8-6667ed013afb-tls-certs\") pod \"kube-auth-proxy-6bcd447999-f4qch\" (UID: \"e692bffa-b34f-4ce9-9db8-6667ed013afb\") " pod="openshift-ingress/kube-auth-proxy-6bcd447999-f4qch" Oct 30 21:27:59.513691 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:59.513662 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 21:27:59.517190 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:59.517153 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 21:27:59.517190 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:59.517153 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-gnpm5\"" Oct 30 21:27:59.582939 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:59.582887 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e692bffa-b34f-4ce9-9db8-6667ed013afb-tmp\") pod \"kube-auth-proxy-6bcd447999-f4qch\" (UID: \"e692bffa-b34f-4ce9-9db8-6667ed013afb\") " pod="openshift-ingress/kube-auth-proxy-6bcd447999-f4qch" Oct 30 21:27:59.583119 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:59.583056 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxz6j\" (UniqueName: \"kubernetes.io/projected/e692bffa-b34f-4ce9-9db8-6667ed013afb-kube-api-access-dxz6j\") pod \"kube-auth-proxy-6bcd447999-f4qch\" (UID: \"e692bffa-b34f-4ce9-9db8-6667ed013afb\") " pod="openshift-ingress/kube-auth-proxy-6bcd447999-f4qch" Oct 30 21:27:59.583119 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:59.583111 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e692bffa-b34f-4ce9-9db8-6667ed013afb-tls-certs\") pod \"kube-auth-proxy-6bcd447999-f4qch\" (UID: \"e692bffa-b34f-4ce9-9db8-6667ed013afb\") " pod="openshift-ingress/kube-auth-proxy-6bcd447999-f4qch" Oct 30 21:27:59.585542 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:59.585515 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e692bffa-b34f-4ce9-9db8-6667ed013afb-tmp\") pod \"kube-auth-proxy-6bcd447999-f4qch\" (UID: \"e692bffa-b34f-4ce9-9db8-6667ed013afb\") " pod="openshift-ingress/kube-auth-proxy-6bcd447999-f4qch" Oct 30 21:27:59.585722 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:59.585701 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e692bffa-b34f-4ce9-9db8-6667ed013afb-tls-certs\") pod \"kube-auth-proxy-6bcd447999-f4qch\" (UID: \"e692bffa-b34f-4ce9-9db8-6667ed013afb\") " pod="openshift-ingress/kube-auth-proxy-6bcd447999-f4qch" Oct 30 21:27:59.647427 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:59.647398 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxz6j\" (UniqueName: \"kubernetes.io/projected/e692bffa-b34f-4ce9-9db8-6667ed013afb-kube-api-access-dxz6j\") pod \"kube-auth-proxy-6bcd447999-f4qch\" (UID: \"e692bffa-b34f-4ce9-9db8-6667ed013afb\") " pod="openshift-ingress/kube-auth-proxy-6bcd447999-f4qch" Oct 30 21:27:59.676383 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:27:59.676346 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6bcd447999-f4qch" Oct 30 21:28:00.058626 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:00.058586 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6bcd447999-f4qch"] Oct 30 21:28:00.061164 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:28:00.061133 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode692bffa_b34f_4ce9_9db8_6667ed013afb.slice/crio-74b66fc96734be915f0f5263e0747c2110e1139ddba0b2c8d15b85bcc391ce8b WatchSource:0}: Error finding container 74b66fc96734be915f0f5263e0747c2110e1139ddba0b2c8d15b85bcc391ce8b: Status 404 returned error can't find the container with id 74b66fc96734be915f0f5263e0747c2110e1139ddba0b2c8d15b85bcc391ce8b Oct 30 21:28:00.148701 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:28:00.148660 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:28:00.646595 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:00.646553 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6bcd447999-f4qch" event={"ID":"e692bffa-b34f-4ce9-9db8-6667ed013afb","Type":"ContainerStarted","Data":"74b66fc96734be915f0f5263e0747c2110e1139ddba0b2c8d15b85bcc391ce8b"} Oct 30 21:28:01.972489 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:01.972412 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-x8dmt"] Oct 30 21:28:01.975600 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:01.975569 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-x8dmt" Oct 30 21:28:02.039710 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:02.039661 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-x8dmt"] Oct 30 21:28:02.087791 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:02.087757 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 21:28:02.109169 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:02.109134 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b593ebf8-d727-4f40-83e8-b8fcfd1e4deb-config-file\") pod \"limitador-limitador-74dfcfb894-x8dmt\" (UID: \"b593ebf8-d727-4f40-83e8-b8fcfd1e4deb\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-x8dmt" Oct 30 21:28:02.109352 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:02.109299 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2x69\" (UniqueName: \"kubernetes.io/projected/b593ebf8-d727-4f40-83e8-b8fcfd1e4deb-kube-api-access-g2x69\") pod \"limitador-limitador-74dfcfb894-x8dmt\" (UID: \"b593ebf8-d727-4f40-83e8-b8fcfd1e4deb\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-x8dmt" Oct 30 21:28:02.210756 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:02.210717 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2x69\" (UniqueName: \"kubernetes.io/projected/b593ebf8-d727-4f40-83e8-b8fcfd1e4deb-kube-api-access-g2x69\") pod \"limitador-limitador-74dfcfb894-x8dmt\" (UID: \"b593ebf8-d727-4f40-83e8-b8fcfd1e4deb\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-x8dmt" Oct 30 21:28:02.210934 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:02.210799 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b593ebf8-d727-4f40-83e8-b8fcfd1e4deb-config-file\") pod \"limitador-limitador-74dfcfb894-x8dmt\" (UID: \"b593ebf8-d727-4f40-83e8-b8fcfd1e4deb\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-x8dmt" Oct 30 21:28:02.211582 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:02.211557 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b593ebf8-d727-4f40-83e8-b8fcfd1e4deb-config-file\") pod \"limitador-limitador-74dfcfb894-x8dmt\" (UID: \"b593ebf8-d727-4f40-83e8-b8fcfd1e4deb\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-x8dmt" Oct 30 21:28:02.226197 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:02.226114 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-x8dmt"] Oct 30 21:28:02.273485 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:02.273453 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2x69\" (UniqueName: \"kubernetes.io/projected/b593ebf8-d727-4f40-83e8-b8fcfd1e4deb-kube-api-access-g2x69\") pod \"limitador-limitador-74dfcfb894-x8dmt\" (UID: \"b593ebf8-d727-4f40-83e8-b8fcfd1e4deb\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-x8dmt" Oct 30 21:28:02.288126 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:02.288077 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-x8dmt" Oct 30 21:28:02.703332 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:02.703298 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-x8dmt"] Oct 30 21:28:02.708156 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:28:02.708125 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb593ebf8_d727_4f40_83e8_b8fcfd1e4deb.slice/crio-ab4fe263baf9a6301f047ec3c27be35b1baee00cc7091e20100bb2d4c9ece750 WatchSource:0}: Error finding container ab4fe263baf9a6301f047ec3c27be35b1baee00cc7091e20100bb2d4c9ece750: Status 404 returned error can't find the container with id ab4fe263baf9a6301f047ec3c27be35b1baee00cc7091e20100bb2d4c9ece750 Oct 30 21:28:03.283032 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:03.282996 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-64f799d7b5-ct7xk"] Oct 30 21:28:03.285416 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:03.285395 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-64f799d7b5-ct7xk" Oct 30 21:28:03.340209 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:03.340180 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-dqljz\"" Oct 30 21:28:03.371985 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:03.371955 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:28:03.390786 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:03.390756 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 30 21:28:03.390786 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:03.390756 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Oct 30 21:28:03.391821 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:03.391800 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-64f799d7b5-ct7xk"] Oct 30 21:28:03.421405 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:03.421338 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f10664cd-868f-4608-bb6c-46bc381720bd-cert\") pod \"jobset-controller-manager-64f799d7b5-ct7xk\" (UID: \"f10664cd-868f-4608-bb6c-46bc381720bd\") " pod="openshift-jobset-operator/jobset-controller-manager-64f799d7b5-ct7xk" Oct 30 21:28:03.421405 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:03.421374 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f10664cd-868f-4608-bb6c-46bc381720bd-manager-config\") pod \"jobset-controller-manager-64f799d7b5-ct7xk\" (UID: \"f10664cd-868f-4608-bb6c-46bc381720bd\") " pod="openshift-jobset-operator/jobset-controller-manager-64f799d7b5-ct7xk" Oct 30 21:28:03.421544 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:03.421411 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f10664cd-868f-4608-bb6c-46bc381720bd-metrics-certs\") pod \"jobset-controller-manager-64f799d7b5-ct7xk\" (UID: \"f10664cd-868f-4608-bb6c-46bc381720bd\") " pod="openshift-jobset-operator/jobset-controller-manager-64f799d7b5-ct7xk" Oct 30 21:28:03.421576 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:03.421531 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-524jr\" (UniqueName: \"kubernetes.io/projected/f10664cd-868f-4608-bb6c-46bc381720bd-kube-api-access-524jr\") pod \"jobset-controller-manager-64f799d7b5-ct7xk\" (UID: \"f10664cd-868f-4608-bb6c-46bc381720bd\") " pod="openshift-jobset-operator/jobset-controller-manager-64f799d7b5-ct7xk" Oct 30 21:28:03.440381 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:03.440346 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Oct 30 21:28:03.440381 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:03.440353 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Oct 30 21:28:03.522043 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:03.522005 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f10664cd-868f-4608-bb6c-46bc381720bd-metrics-certs\") pod \"jobset-controller-manager-64f799d7b5-ct7xk\" (UID: \"f10664cd-868f-4608-bb6c-46bc381720bd\") " pod="openshift-jobset-operator/jobset-controller-manager-64f799d7b5-ct7xk" Oct 30 21:28:03.522252 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:03.522071 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-524jr\" (UniqueName: \"kubernetes.io/projected/f10664cd-868f-4608-bb6c-46bc381720bd-kube-api-access-524jr\") pod \"jobset-controller-manager-64f799d7b5-ct7xk\" (UID: \"f10664cd-868f-4608-bb6c-46bc381720bd\") " pod="openshift-jobset-operator/jobset-controller-manager-64f799d7b5-ct7xk" Oct 30 21:28:03.522252 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:03.522120 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f10664cd-868f-4608-bb6c-46bc381720bd-cert\") pod \"jobset-controller-manager-64f799d7b5-ct7xk\" (UID: \"f10664cd-868f-4608-bb6c-46bc381720bd\") " pod="openshift-jobset-operator/jobset-controller-manager-64f799d7b5-ct7xk" Oct 30 21:28:03.522252 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:03.522144 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f10664cd-868f-4608-bb6c-46bc381720bd-manager-config\") pod \"jobset-controller-manager-64f799d7b5-ct7xk\" (UID: \"f10664cd-868f-4608-bb6c-46bc381720bd\") " pod="openshift-jobset-operator/jobset-controller-manager-64f799d7b5-ct7xk" Oct 30 21:28:03.522868 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:03.522847 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f10664cd-868f-4608-bb6c-46bc381720bd-manager-config\") pod \"jobset-controller-manager-64f799d7b5-ct7xk\" (UID: \"f10664cd-868f-4608-bb6c-46bc381720bd\") " pod="openshift-jobset-operator/jobset-controller-manager-64f799d7b5-ct7xk" Oct 30 21:28:03.524555 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:03.524525 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f10664cd-868f-4608-bb6c-46bc381720bd-metrics-certs\") pod \"jobset-controller-manager-64f799d7b5-ct7xk\" (UID: \"f10664cd-868f-4608-bb6c-46bc381720bd\") " pod="openshift-jobset-operator/jobset-controller-manager-64f799d7b5-ct7xk" Oct 30 21:28:03.524640 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:03.524557 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f10664cd-868f-4608-bb6c-46bc381720bd-cert\") pod \"jobset-controller-manager-64f799d7b5-ct7xk\" (UID: \"f10664cd-868f-4608-bb6c-46bc381720bd\") " pod="openshift-jobset-operator/jobset-controller-manager-64f799d7b5-ct7xk" Oct 30 21:28:03.594520 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:03.594482 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-524jr\" (UniqueName: \"kubernetes.io/projected/f10664cd-868f-4608-bb6c-46bc381720bd-kube-api-access-524jr\") pod \"jobset-controller-manager-64f799d7b5-ct7xk\" (UID: \"f10664cd-868f-4608-bb6c-46bc381720bd\") " pod="openshift-jobset-operator/jobset-controller-manager-64f799d7b5-ct7xk" Oct 30 21:28:03.603391 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:03.603353 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-64f799d7b5-ct7xk" Oct 30 21:28:03.671032 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:03.670954 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6bcd447999-f4qch" event={"ID":"e692bffa-b34f-4ce9-9db8-6667ed013afb","Type":"ContainerStarted","Data":"e3adc96875b98c86d0c12839dfc9dced2095d9ec37d123df24e35cc7a53e3b26"} Oct 30 21:28:03.674175 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:03.674145 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-x8dmt" event={"ID":"b593ebf8-d727-4f40-83e8-b8fcfd1e4deb","Type":"ContainerStarted","Data":"ab4fe263baf9a6301f047ec3c27be35b1baee00cc7091e20100bb2d4c9ece750"} Oct 30 21:28:03.675787 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:03.675757 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fm72c" event={"ID":"8b9b71d4-264b-4666-bfe3-ff0dcffafc85","Type":"ContainerStarted","Data":"040f3db81d98d40c25b0f68bbd5d5f6528654710cb5dc54a3d5ac71b2a02c1e0"} Oct 30 21:28:03.805589 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:03.805292 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6bcd447999-f4qch" podStartSLOduration=1.719558761 podStartE2EDuration="4.805276967s" totalImagesPullingTime="3.085718206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:03.804274797 +0000 UTC m=+267.546795732" watchObservedRunningTime="2026-10-30 21:28:03.805276967 +0000 UTC m=+267.547797901" Oct 30 21:28:03.930123 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:03.930020 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fm72c" podStartSLOduration=2.889526441 podStartE2EDuration="8.930003976s" totalImagesPullingTime="6.040477535s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:03.929629505 +0000 UTC m=+267.672150441" watchObservedRunningTime="2026-10-30 21:28:03.930003976 +0000 UTC m=+267.672524952" Oct 30 21:28:03.997709 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:03.997670 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-64f799d7b5-ct7xk"] Oct 30 21:28:04.000796 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:28:04.000770 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf10664cd_868f_4608_bb6c_46bc381720bd.slice/crio-00cf7f29bc1f0edbf03662d56087e477dad1b353385f739317a2a974d128c0a2 WatchSource:0}: Error finding container 00cf7f29bc1f0edbf03662d56087e477dad1b353385f739317a2a974d128c0a2: Status 404 returned error can't find the container with id 00cf7f29bc1f0edbf03662d56087e477dad1b353385f739317a2a974d128c0a2 Oct 30 21:28:04.681548 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:04.681506 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-64f799d7b5-ct7xk" event={"ID":"f10664cd-868f-4608-bb6c-46bc381720bd","Type":"ContainerStarted","Data":"00cf7f29bc1f0edbf03662d56087e477dad1b353385f739317a2a974d128c0a2"} Oct 30 21:28:06.697771 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:06.696604 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-x8dmt" event={"ID":"b593ebf8-d727-4f40-83e8-b8fcfd1e4deb","Type":"ContainerStarted","Data":"26b5afadff3f886b80f1c25c50c434caab1a37c4131eb8f4b2a851b959279119"} Oct 30 21:28:06.697771 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:06.697639 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-x8dmt" Oct 30 21:28:06.816243 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:06.816176 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-x8dmt" podStartSLOduration=2.100675569 podStartE2EDuration="5.816153611s" totalImagesPullingTime="3.715478042s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:06.814881058 +0000 UTC m=+270.557401993" watchObservedRunningTime="2026-10-30 21:28:06.816153611 +0000 UTC m=+270.558674549" Oct 30 21:28:08.706943 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:08.706889 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-64f799d7b5-ct7xk" event={"ID":"f10664cd-868f-4608-bb6c-46bc381720bd","Type":"ContainerStarted","Data":"90a851bae9118d60fe8c6ef3e1762d3db09cb7e23c88638c33e1ff8635c8c572"} Oct 30 21:28:08.707327 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:08.707144 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-64f799d7b5-ct7xk" Oct 30 21:28:08.824444 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:08.824386 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-64f799d7b5-ct7xk" podStartSLOduration=1.759813112 podStartE2EDuration="5.8243695s" totalImagesPullingTime="4.064556388s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:08.824064608 +0000 UTC m=+272.566585543" watchObservedRunningTime="2026-10-30 21:28:08.8243695 +0000 UTC m=+272.566890434" Oct 30 21:28:18.707804 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:18.707764 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-x8dmt" Oct 30 21:28:19.716754 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:19.716724 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-64f799d7b5-ct7xk" Oct 30 21:28:20.824710 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:20.824673 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-x8dmt"] Oct 30 21:28:20.825134 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:20.824874 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-x8dmt" podUID="b593ebf8-d727-4f40-83e8-b8fcfd1e4deb" containerName="limitador" containerID="cri-o://26b5afadff3f886b80f1c25c50c434caab1a37c4131eb8f4b2a851b959279119" gracePeriod=30 Oct 30 21:28:21.366132 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:21.366106 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-x8dmt" Oct 30 21:28:21.471769 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:21.471738 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b593ebf8-d727-4f40-83e8-b8fcfd1e4deb-kube-api-access-g2x69\" (UniqueName: \"kubernetes.io/projected/b593ebf8-d727-4f40-83e8-b8fcfd1e4deb-kube-api-access-g2x69\") pod \"b593ebf8-d727-4f40-83e8-b8fcfd1e4deb\" (UID: \"b593ebf8-d727-4f40-83e8-b8fcfd1e4deb\") " Oct 30 21:28:21.471993 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:21.471922 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b593ebf8-d727-4f40-83e8-b8fcfd1e4deb-config-file\" (UniqueName: \"kubernetes.io/configmap/b593ebf8-d727-4f40-83e8-b8fcfd1e4deb-config-file\") pod \"b593ebf8-d727-4f40-83e8-b8fcfd1e4deb\" (UID: \"b593ebf8-d727-4f40-83e8-b8fcfd1e4deb\") " Oct 30 21:28:21.472287 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:21.472260 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b593ebf8-d727-4f40-83e8-b8fcfd1e4deb-config-file" pod "b593ebf8-d727-4f40-83e8-b8fcfd1e4deb" (UID: "b593ebf8-d727-4f40-83e8-b8fcfd1e4deb"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:28:21.474032 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:21.474004 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b593ebf8-d727-4f40-83e8-b8fcfd1e4deb-kube-api-access-g2x69" pod "b593ebf8-d727-4f40-83e8-b8fcfd1e4deb" (UID: "b593ebf8-d727-4f40-83e8-b8fcfd1e4deb"). InnerVolumeSpecName "kube-api-access-g2x69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:28:21.572842 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:21.572807 5748 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b593ebf8-d727-4f40-83e8-b8fcfd1e4deb-config-file\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:28:21.572842 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:21.572846 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2x69\" (UniqueName: \"kubernetes.io/projected/b593ebf8-d727-4f40-83e8-b8fcfd1e4deb-kube-api-access-g2x69\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:28:21.770818 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:21.770725 5748 generic.go:356] "Generic (PLEG): container finished" podID="b593ebf8-d727-4f40-83e8-b8fcfd1e4deb" containerID="26b5afadff3f886b80f1c25c50c434caab1a37c4131eb8f4b2a851b959279119" exitCode=0 Oct 30 21:28:21.770818 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:21.770771 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-x8dmt" event={"ID":"b593ebf8-d727-4f40-83e8-b8fcfd1e4deb","Type":"ContainerDied","Data":"26b5afadff3f886b80f1c25c50c434caab1a37c4131eb8f4b2a851b959279119"} Oct 30 21:28:21.770818 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:21.770778 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-x8dmt" Oct 30 21:28:21.770818 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:21.770813 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-x8dmt" event={"ID":"b593ebf8-d727-4f40-83e8-b8fcfd1e4deb","Type":"ContainerDied","Data":"ab4fe263baf9a6301f047ec3c27be35b1baee00cc7091e20100bb2d4c9ece750"} Oct 30 21:28:21.771194 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:21.770835 5748 scope.go:122] "RemoveContainer" containerID="26b5afadff3f886b80f1c25c50c434caab1a37c4131eb8f4b2a851b959279119" Oct 30 21:28:21.779402 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:21.779377 5748 scope.go:122] "RemoveContainer" containerID="26b5afadff3f886b80f1c25c50c434caab1a37c4131eb8f4b2a851b959279119" Oct 30 21:28:21.779669 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:28:21.779648 5748 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26b5afadff3f886b80f1c25c50c434caab1a37c4131eb8f4b2a851b959279119\": container with ID starting with 26b5afadff3f886b80f1c25c50c434caab1a37c4131eb8f4b2a851b959279119 not found: ID does not exist" containerID="26b5afadff3f886b80f1c25c50c434caab1a37c4131eb8f4b2a851b959279119" Oct 30 21:28:21.779740 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:21.779683 5748 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26b5afadff3f886b80f1c25c50c434caab1a37c4131eb8f4b2a851b959279119"} err="failed to get container status \"26b5afadff3f886b80f1c25c50c434caab1a37c4131eb8f4b2a851b959279119\": rpc error: code = NotFound desc = could not find container \"26b5afadff3f886b80f1c25c50c434caab1a37c4131eb8f4b2a851b959279119\": container with ID starting with 26b5afadff3f886b80f1c25c50c434caab1a37c4131eb8f4b2a851b959279119 not found: ID does not exist" Oct 30 21:28:21.943558 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:21.941951 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-x8dmt"] Oct 30 21:28:21.991316 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:21.991278 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-x8dmt"] Oct 30 21:28:22.415250 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:22.415212 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b593ebf8-d727-4f40-83e8-b8fcfd1e4deb" path="/var/lib/kubelet/pods/b593ebf8-d727-4f40-83e8-b8fcfd1e4deb/volumes" Oct 30 21:28:30.144368 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:28:30.144327 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:28:36.436428 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:36.436399 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j4v8r_3df036ff-1c00-4ce2-9e86-8acbeedf6c82/ovn-acl-logging/0.log" Oct 30 21:28:36.436836 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:28:36.436509 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j4v8r_3df036ff-1c00-4ce2-9e86-8acbeedf6c82/ovn-acl-logging/0.log" Oct 30 21:29:00.144337 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:29:00.144298 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:29:19.071797 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:29:19.071755 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-tkm9r"] Oct 30 21:29:19.072431 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:29:19.072399 5748 cpu_manager.go:414] "containerMap: removing container" podUID="b593ebf8-d727-4f40-83e8-b8fcfd1e4deb" containerName="limitador" Oct 30 21:29:19.072431 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:29:19.072420 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b593ebf8-d727-4f40-83e8-b8fcfd1e4deb" containerName="limitador" Oct 30 21:29:19.072550 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:29:19.072508 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="b593ebf8-d727-4f40-83e8-b8fcfd1e4deb" containerName="limitador" Oct 30 21:29:19.074623 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:29:19.074602 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tkm9r" Oct 30 21:29:19.134050 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:29:19.134016 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-tkm9r"] Oct 30 21:29:19.180427 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:29:19.180392 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 21:29:19.180623 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:29:19.180515 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 30 21:29:19.180623 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:29:19.180523 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-m9khr\"" Oct 30 21:29:19.180623 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:29:19.180578 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 21:29:19.183285 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:29:19.183259 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/71c9e869-a17e-42f3-86b1-12d1199ff2d4-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-tkm9r\" (UID: \"71c9e869-a17e-42f3-86b1-12d1199ff2d4\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tkm9r" Oct 30 21:29:19.183402 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:29:19.183293 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9fnh\" (UniqueName: \"kubernetes.io/projected/71c9e869-a17e-42f3-86b1-12d1199ff2d4-kube-api-access-g9fnh\") pod \"llmisvc-controller-manager-869d5dcc5d-tkm9r\" (UID: \"71c9e869-a17e-42f3-86b1-12d1199ff2d4\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tkm9r" Oct 30 21:29:19.188712 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:29:19.188690 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-nklnt"] Oct 30 21:29:19.191160 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:29:19.191134 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-nklnt" Oct 30 21:29:19.245467 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:29:19.245431 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Oct 30 21:29:19.245792 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:29:19.245559 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-v47hp\"" Oct 30 21:29:19.251176 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:29:19.251145 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-nklnt"] Oct 30 21:29:19.284042 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:29:19.284005 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/71c9e869-a17e-42f3-86b1-12d1199ff2d4-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-tkm9r\" (UID: \"71c9e869-a17e-42f3-86b1-12d1199ff2d4\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tkm9r" Oct 30 21:29:19.284042 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:29:19.284041 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9fnh\" (UniqueName: \"kubernetes.io/projected/71c9e869-a17e-42f3-86b1-12d1199ff2d4-kube-api-access-g9fnh\") pod \"llmisvc-controller-manager-869d5dcc5d-tkm9r\" (UID: \"71c9e869-a17e-42f3-86b1-12d1199ff2d4\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tkm9r" Oct 30 21:29:19.284295 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:29:19.284068 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t9lt\" (UniqueName: \"kubernetes.io/projected/62a7105e-4256-4b8d-8117-25d9a9c8a506-kube-api-access-8t9lt\") pod \"model-serving-api-75bd89fdfc-nklnt\" (UID: \"62a7105e-4256-4b8d-8117-25d9a9c8a506\") " pod="opendatahub/model-serving-api-75bd89fdfc-nklnt" Oct 30 21:29:19.284360 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:29:19.284286 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62a7105e-4256-4b8d-8117-25d9a9c8a506-tls-certs\") pod \"model-serving-api-75bd89fdfc-nklnt\" (UID: \"62a7105e-4256-4b8d-8117-25d9a9c8a506\") " pod="opendatahub/model-serving-api-75bd89fdfc-nklnt" Oct 30 21:29:19.286591 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:29:19.286563 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/71c9e869-a17e-42f3-86b1-12d1199ff2d4-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-tkm9r\" (UID: \"71c9e869-a17e-42f3-86b1-12d1199ff2d4\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tkm9r" Oct 30 21:29:19.346576 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:29:19.346456 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9fnh\" (UniqueName: \"kubernetes.io/projected/71c9e869-a17e-42f3-86b1-12d1199ff2d4-kube-api-access-g9fnh\") pod \"llmisvc-controller-manager-869d5dcc5d-tkm9r\" (UID: \"71c9e869-a17e-42f3-86b1-12d1199ff2d4\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tkm9r" Oct 30 21:29:19.385256 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:29:19.385221 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t9lt\" (UniqueName: \"kubernetes.io/projected/62a7105e-4256-4b8d-8117-25d9a9c8a506-kube-api-access-8t9lt\") pod \"model-serving-api-75bd89fdfc-nklnt\" (UID: \"62a7105e-4256-4b8d-8117-25d9a9c8a506\") " pod="opendatahub/model-serving-api-75bd89fdfc-nklnt" Oct 30 21:29:19.385443 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:29:19.385314 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62a7105e-4256-4b8d-8117-25d9a9c8a506-tls-certs\") pod \"model-serving-api-75bd89fdfc-nklnt\" (UID: \"62a7105e-4256-4b8d-8117-25d9a9c8a506\") " pod="opendatahub/model-serving-api-75bd89fdfc-nklnt" Oct 30 21:29:19.385443 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:29:19.385428 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tkm9r" Oct 30 21:29:19.387835 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:29:19.387815 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62a7105e-4256-4b8d-8117-25d9a9c8a506-tls-certs\") pod \"model-serving-api-75bd89fdfc-nklnt\" (UID: \"62a7105e-4256-4b8d-8117-25d9a9c8a506\") " pod="opendatahub/model-serving-api-75bd89fdfc-nklnt" Oct 30 21:29:19.454230 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:29:19.453319 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t9lt\" (UniqueName: \"kubernetes.io/projected/62a7105e-4256-4b8d-8117-25d9a9c8a506-kube-api-access-8t9lt\") pod \"model-serving-api-75bd89fdfc-nklnt\" (UID: \"62a7105e-4256-4b8d-8117-25d9a9c8a506\") " pod="opendatahub/model-serving-api-75bd89fdfc-nklnt" Oct 30 21:29:19.502921 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:29:19.502865 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-nklnt" Oct 30 21:29:19.800184 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:29:19.800160 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-tkm9r"] Oct 30 21:29:19.801771 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:29:19.801710 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod71c9e869_a17e_42f3_86b1_12d1199ff2d4.slice/crio-a96814e6849e1c826cf98cf01b349a491d530da57e7f491a873e76ad6962c0dd WatchSource:0}: Error finding container a96814e6849e1c826cf98cf01b349a491d530da57e7f491a873e76ad6962c0dd: Status 404 returned error can't find the container with id a96814e6849e1c826cf98cf01b349a491d530da57e7f491a873e76ad6962c0dd Oct 30 21:29:19.803101 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:29:19.803078 5748 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:29:19.886001 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:29:19.885957 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-nklnt"] Oct 30 21:29:19.888560 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:29:19.888529 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62a7105e_4256_4b8d_8117_25d9a9c8a506.slice/crio-6f8e809fe67f7afd07e998dc9314a2b38fa229b6f86200f44768d38c348cc564 WatchSource:0}: Error finding container 6f8e809fe67f7afd07e998dc9314a2b38fa229b6f86200f44768d38c348cc564: Status 404 returned error can't find the container with id 6f8e809fe67f7afd07e998dc9314a2b38fa229b6f86200f44768d38c348cc564 Oct 30 21:29:20.020218 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:29:20.020183 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-nklnt" event={"ID":"62a7105e-4256-4b8d-8117-25d9a9c8a506","Type":"ContainerStarted","Data":"6f8e809fe67f7afd07e998dc9314a2b38fa229b6f86200f44768d38c348cc564"} Oct 30 21:29:20.021303 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:29:20.021281 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tkm9r" event={"ID":"71c9e869-a17e-42f3-86b1-12d1199ff2d4","Type":"ContainerStarted","Data":"a96814e6849e1c826cf98cf01b349a491d530da57e7f491a873e76ad6962c0dd"} Oct 30 21:29:23.039560 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:29:23.039522 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-nklnt" event={"ID":"62a7105e-4256-4b8d-8117-25d9a9c8a506","Type":"ContainerStarted","Data":"2e36469898bdc1712276fd5042ec851d2e7ca682e0b53a22dc064e82792a581d"} Oct 30 21:29:23.040054 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:29:23.039630 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-nklnt" Oct 30 21:29:23.041221 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:29:23.041197 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tkm9r" event={"ID":"71c9e869-a17e-42f3-86b1-12d1199ff2d4","Type":"ContainerStarted","Data":"2eda0ebba361532f0faed7a5701662df9beda3ef883184c32559651172530f98"} Oct 30 21:29:23.041301 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:29:23.041243 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tkm9r" Oct 30 21:29:23.160364 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:29:23.160308 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-nklnt" podStartSLOduration=2.038720234 podStartE2EDuration="4.160291738s" totalImagesPullingTime="2.121571504s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:29:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:29:23.159490631 +0000 UTC m=+346.902011568" watchObservedRunningTime="2026-10-30 21:29:23.160291738 +0000 UTC m=+346.902812673" Oct 30 21:29:23.278380 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:29:23.278324 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tkm9r" podStartSLOduration=1.697752501 podStartE2EDuration="4.27831101s" totalImagesPullingTime="2.580558509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:29:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:29:23.27783047 +0000 UTC m=+347.020351408" watchObservedRunningTime="2026-10-30 21:29:23.27831101 +0000 UTC m=+347.020831942" Oct 30 21:29:30.144803 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:29:30.144762 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:29:34.051052 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:29:34.051017 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-nklnt" Oct 30 21:29:54.048268 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:29:54.048189 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tkm9r" Oct 30 21:30:00.144861 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:30:00.144816 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:30:30.143523 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:30:30.143485 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:30:44.216801 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:30:44.216764 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-zq226"] Oct 30 21:30:44.220148 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:30:44.220129 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-zq226" Oct 30 21:30:44.275729 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:30:44.275697 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bltd\" (UniqueName: \"kubernetes.io/projected/d611ba76-37b9-4cd6-aac5-14dfa4cf136b-kube-api-access-5bltd\") pod \"llmisvc-controller-manager-6466c554d8-zq226\" (UID: \"d611ba76-37b9-4cd6-aac5-14dfa4cf136b\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-zq226" Oct 30 21:30:44.275936 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:30:44.275764 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d611ba76-37b9-4cd6-aac5-14dfa4cf136b-cert\") pod \"llmisvc-controller-manager-6466c554d8-zq226\" (UID: \"d611ba76-37b9-4cd6-aac5-14dfa4cf136b\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-zq226" Oct 30 21:30:44.279554 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:30:44.279525 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-zq226"] Oct 30 21:30:44.376557 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:30:44.376520 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d611ba76-37b9-4cd6-aac5-14dfa4cf136b-cert\") pod \"llmisvc-controller-manager-6466c554d8-zq226\" (UID: \"d611ba76-37b9-4cd6-aac5-14dfa4cf136b\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-zq226" Oct 30 21:30:44.376737 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:30:44.376576 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bltd\" (UniqueName: \"kubernetes.io/projected/d611ba76-37b9-4cd6-aac5-14dfa4cf136b-kube-api-access-5bltd\") pod \"llmisvc-controller-manager-6466c554d8-zq226\" (UID: \"d611ba76-37b9-4cd6-aac5-14dfa4cf136b\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-zq226" Oct 30 21:30:44.379066 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:30:44.379037 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d611ba76-37b9-4cd6-aac5-14dfa4cf136b-cert\") pod \"llmisvc-controller-manager-6466c554d8-zq226\" (UID: \"d611ba76-37b9-4cd6-aac5-14dfa4cf136b\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-zq226" Oct 30 21:30:44.438987 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:30:44.438945 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bltd\" (UniqueName: \"kubernetes.io/projected/d611ba76-37b9-4cd6-aac5-14dfa4cf136b-kube-api-access-5bltd\") pod \"llmisvc-controller-manager-6466c554d8-zq226\" (UID: \"d611ba76-37b9-4cd6-aac5-14dfa4cf136b\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-zq226" Oct 30 21:30:44.532820 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:30:44.532741 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-zq226" Oct 30 21:30:44.905044 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:30:44.905017 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-zq226"] Oct 30 21:30:44.907691 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:30:44.907660 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd611ba76_37b9_4cd6_aac5_14dfa4cf136b.slice/crio-384e9a73de50e02b6cde3bdd8a910ef9b1d6f5890c733ec9dfc59e2d84db58d4 WatchSource:0}: Error finding container 384e9a73de50e02b6cde3bdd8a910ef9b1d6f5890c733ec9dfc59e2d84db58d4: Status 404 returned error can't find the container with id 384e9a73de50e02b6cde3bdd8a910ef9b1d6f5890c733ec9dfc59e2d84db58d4 Oct 30 21:30:45.401824 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:30:45.401790 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-zq226" event={"ID":"d611ba76-37b9-4cd6-aac5-14dfa4cf136b","Type":"ContainerStarted","Data":"384e9a73de50e02b6cde3bdd8a910ef9b1d6f5890c733ec9dfc59e2d84db58d4"} Oct 30 21:30:46.408222 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:30:46.408184 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-zq226" event={"ID":"d611ba76-37b9-4cd6-aac5-14dfa4cf136b","Type":"ContainerStarted","Data":"c845ed4049e0edf64b4a8335829e8b4dc8a2c93ea3aaeb8d7a69f298663084e9"} Oct 30 21:30:46.408662 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:30:46.408300 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-zq226" Oct 30 21:30:46.527138 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:30:46.527075 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-zq226" podStartSLOduration=1.779032341 podStartE2EDuration="2.527060748s" totalImagesPullingTime="748.028407ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:46.526351922 +0000 UTC m=+430.268872858" watchObservedRunningTime="2026-10-30 21:30:46.527060748 +0000 UTC m=+430.269581691" Oct 30 21:31:00.144484 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:31:00.144443 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:31:17.414395 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:31:17.414359 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-zq226" Oct 30 21:31:17.584258 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:31:17.584225 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-tkm9r"] Oct 30 21:31:17.584594 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:31:17.584441 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tkm9r" podUID="71c9e869-a17e-42f3-86b1-12d1199ff2d4" containerName="manager" containerID="cri-o://2eda0ebba361532f0faed7a5701662df9beda3ef883184c32559651172530f98" gracePeriod=30 Oct 30 21:31:17.828931 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:31:17.828887 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tkm9r" Oct 30 21:31:17.983588 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:31:17.983505 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71c9e869-a17e-42f3-86b1-12d1199ff2d4-cert\" (UniqueName: \"kubernetes.io/secret/71c9e869-a17e-42f3-86b1-12d1199ff2d4-cert\") pod \"71c9e869-a17e-42f3-86b1-12d1199ff2d4\" (UID: \"71c9e869-a17e-42f3-86b1-12d1199ff2d4\") " Oct 30 21:31:17.983588 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:31:17.983576 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71c9e869-a17e-42f3-86b1-12d1199ff2d4-kube-api-access-g9fnh\" (UniqueName: \"kubernetes.io/projected/71c9e869-a17e-42f3-86b1-12d1199ff2d4-kube-api-access-g9fnh\") pod \"71c9e869-a17e-42f3-86b1-12d1199ff2d4\" (UID: \"71c9e869-a17e-42f3-86b1-12d1199ff2d4\") " Oct 30 21:31:17.985809 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:31:17.985776 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71c9e869-a17e-42f3-86b1-12d1199ff2d4-cert" pod "71c9e869-a17e-42f3-86b1-12d1199ff2d4" (UID: "71c9e869-a17e-42f3-86b1-12d1199ff2d4"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:31:17.985885 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:31:17.985840 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71c9e869-a17e-42f3-86b1-12d1199ff2d4-kube-api-access-g9fnh" pod "71c9e869-a17e-42f3-86b1-12d1199ff2d4" (UID: "71c9e869-a17e-42f3-86b1-12d1199ff2d4"). InnerVolumeSpecName "kube-api-access-g9fnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:31:18.084501 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:31:18.084465 5748 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/71c9e869-a17e-42f3-86b1-12d1199ff2d4-cert\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:31:18.084501 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:31:18.084498 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9fnh\" (UniqueName: \"kubernetes.io/projected/71c9e869-a17e-42f3-86b1-12d1199ff2d4-kube-api-access-g9fnh\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:31:18.550353 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:31:18.550314 5748 generic.go:356] "Generic (PLEG): container finished" podID="71c9e869-a17e-42f3-86b1-12d1199ff2d4" containerID="2eda0ebba361532f0faed7a5701662df9beda3ef883184c32559651172530f98" exitCode=0 Oct 30 21:31:18.550853 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:31:18.550382 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tkm9r" Oct 30 21:31:18.550853 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:31:18.550392 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tkm9r" event={"ID":"71c9e869-a17e-42f3-86b1-12d1199ff2d4","Type":"ContainerDied","Data":"2eda0ebba361532f0faed7a5701662df9beda3ef883184c32559651172530f98"} Oct 30 21:31:18.550853 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:31:18.550463 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tkm9r" event={"ID":"71c9e869-a17e-42f3-86b1-12d1199ff2d4","Type":"ContainerDied","Data":"a96814e6849e1c826cf98cf01b349a491d530da57e7f491a873e76ad6962c0dd"} Oct 30 21:31:18.550853 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:31:18.550494 5748 scope.go:122] "RemoveContainer" containerID="2eda0ebba361532f0faed7a5701662df9beda3ef883184c32559651172530f98" Oct 30 21:31:18.559243 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:31:18.559225 5748 scope.go:122] "RemoveContainer" containerID="2eda0ebba361532f0faed7a5701662df9beda3ef883184c32559651172530f98" Oct 30 21:31:18.559558 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:31:18.559537 5748 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2eda0ebba361532f0faed7a5701662df9beda3ef883184c32559651172530f98\": container with ID starting with 2eda0ebba361532f0faed7a5701662df9beda3ef883184c32559651172530f98 not found: ID does not exist" containerID="2eda0ebba361532f0faed7a5701662df9beda3ef883184c32559651172530f98" Oct 30 21:31:18.559630 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:31:18.559569 5748 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2eda0ebba361532f0faed7a5701662df9beda3ef883184c32559651172530f98"} err="failed to get container status \"2eda0ebba361532f0faed7a5701662df9beda3ef883184c32559651172530f98\": rpc error: code = NotFound desc = could not find container \"2eda0ebba361532f0faed7a5701662df9beda3ef883184c32559651172530f98\": container with ID starting with 2eda0ebba361532f0faed7a5701662df9beda3ef883184c32559651172530f98 not found: ID does not exist" Oct 30 21:31:18.723710 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:31:18.723630 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-tkm9r"] Oct 30 21:31:18.774200 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:31:18.774166 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-tkm9r"] Oct 30 21:31:20.415401 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:31:20.415360 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71c9e869-a17e-42f3-86b1-12d1199ff2d4" path="/var/lib/kubelet/pods/71c9e869-a17e-42f3-86b1-12d1199ff2d4/volumes" Oct 30 21:31:30.144997 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:31:30.144949 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:32:00.144361 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:32:00.144324 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:32:10.802121 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:32:10.802088 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t7djv"] Oct 30 21:32:10.802624 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:32:10.802453 5748 cpu_manager.go:414] "containerMap: removing container" podUID="71c9e869-a17e-42f3-86b1-12d1199ff2d4" containerName="manager" Oct 30 21:32:10.802624 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:32:10.802463 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71c9e869-a17e-42f3-86b1-12d1199ff2d4" containerName="manager" Oct 30 21:32:10.802624 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:32:10.802545 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="71c9e869-a17e-42f3-86b1-12d1199ff2d4" containerName="manager" Oct 30 21:32:10.805688 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:32:10.805670 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t7djv" Oct 30 21:32:10.862804 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:32:10.862777 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 21:32:10.862993 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:32:10.862838 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-lfxgk\"" Oct 30 21:32:10.912397 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:32:10.912363 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t7djv"] Oct 30 21:32:10.953596 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:32:10.953553 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/818838b2-f54f-4591-b855-73014d18a06e-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-t7djv\" (UID: \"818838b2-f54f-4591-b855-73014d18a06e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t7djv" Oct 30 21:32:10.953596 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:32:10.953603 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/818838b2-f54f-4591-b855-73014d18a06e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-t7djv\" (UID: \"818838b2-f54f-4591-b855-73014d18a06e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t7djv" Oct 30 21:32:10.953866 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:32:10.953702 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhw6f\" (UniqueName: \"kubernetes.io/projected/818838b2-f54f-4591-b855-73014d18a06e-kube-api-access-bhw6f\") pod \"maas-default-gateway-openshift-default-75d7c785f-t7djv\" (UID: \"818838b2-f54f-4591-b855-73014d18a06e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t7djv" Oct 30 21:32:10.953866 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:32:10.953758 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/818838b2-f54f-4591-b855-73014d18a06e-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-t7djv\" (UID: \"818838b2-f54f-4591-b855-73014d18a06e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t7djv" Oct 30 21:32:10.953866 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:32:10.953843 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/818838b2-f54f-4591-b855-73014d18a06e-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-t7djv\" (UID: \"818838b2-f54f-4591-b855-73014d18a06e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t7djv" Oct 30 21:32:10.954066 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:32:10.953871 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/818838b2-f54f-4591-b855-73014d18a06e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-t7djv\" (UID: \"818838b2-f54f-4591-b855-73014d18a06e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t7djv" Oct 30 21:32:10.954066 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:32:10.953971 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/818838b2-f54f-4591-b855-73014d18a06e-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-t7djv\" (UID: \"818838b2-f54f-4591-b855-73014d18a06e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t7djv" Oct 30 21:32:10.954066 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:32:10.954009 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/818838b2-f54f-4591-b855-73014d18a06e-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-t7djv\" (UID: \"818838b2-f54f-4591-b855-73014d18a06e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t7djv" Oct 30 21:32:10.954206 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:32:10.954082 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/818838b2-f54f-4591-b855-73014d18a06e-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-t7djv\" (UID: \"818838b2-f54f-4591-b855-73014d18a06e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t7djv" Oct 30 21:32:11.054884 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:32:11.054799 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/818838b2-f54f-4591-b855-73014d18a06e-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-t7djv\" (UID: \"818838b2-f54f-4591-b855-73014d18a06e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t7djv" Oct 30 21:32:11.054884 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:32:11.054837 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/818838b2-f54f-4591-b855-73014d18a06e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-t7djv\" (UID: \"818838b2-f54f-4591-b855-73014d18a06e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t7djv" Oct 30 21:32:11.054884 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:32:11.054857 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhw6f\" (UniqueName: \"kubernetes.io/projected/818838b2-f54f-4591-b855-73014d18a06e-kube-api-access-bhw6f\") pod \"maas-default-gateway-openshift-default-75d7c785f-t7djv\" (UID: \"818838b2-f54f-4591-b855-73014d18a06e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t7djv" Oct 30 21:32:11.055190 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:32:11.054890 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/818838b2-f54f-4591-b855-73014d18a06e-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-t7djv\" (UID: \"818838b2-f54f-4591-b855-73014d18a06e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t7djv" Oct 30 21:32:11.055190 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:32:11.054970 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/818838b2-f54f-4591-b855-73014d18a06e-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-t7djv\" (UID: \"818838b2-f54f-4591-b855-73014d18a06e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t7djv" Oct 30 21:32:11.055190 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:32:11.054993 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/818838b2-f54f-4591-b855-73014d18a06e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-t7djv\" (UID: \"818838b2-f54f-4591-b855-73014d18a06e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t7djv" Oct 30 21:32:11.055190 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:32:11.055027 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/818838b2-f54f-4591-b855-73014d18a06e-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-t7djv\" (UID: \"818838b2-f54f-4591-b855-73014d18a06e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t7djv" Oct 30 21:32:11.055190 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:32:11.055053 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/818838b2-f54f-4591-b855-73014d18a06e-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-t7djv\" (UID: \"818838b2-f54f-4591-b855-73014d18a06e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t7djv" Oct 30 21:32:11.055190 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:32:11.055080 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/818838b2-f54f-4591-b855-73014d18a06e-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-t7djv\" (UID: \"818838b2-f54f-4591-b855-73014d18a06e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t7djv" Oct 30 21:32:11.055573 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:32:11.055313 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/818838b2-f54f-4591-b855-73014d18a06e-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-t7djv\" (UID: \"818838b2-f54f-4591-b855-73014d18a06e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t7djv" Oct 30 21:32:11.055573 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:32:11.055400 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/818838b2-f54f-4591-b855-73014d18a06e-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-t7djv\" (UID: \"818838b2-f54f-4591-b855-73014d18a06e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t7djv" Oct 30 21:32:11.055573 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:32:11.055493 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/818838b2-f54f-4591-b855-73014d18a06e-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-t7djv\" (UID: \"818838b2-f54f-4591-b855-73014d18a06e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t7djv" Oct 30 21:32:11.055679 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:32:11.055585 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/818838b2-f54f-4591-b855-73014d18a06e-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-t7djv\" (UID: \"818838b2-f54f-4591-b855-73014d18a06e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t7djv" Oct 30 21:32:11.055679 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:32:11.055625 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/818838b2-f54f-4591-b855-73014d18a06e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-t7djv\" (UID: \"818838b2-f54f-4591-b855-73014d18a06e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t7djv" Oct 30 21:32:11.057319 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:32:11.057296 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/818838b2-f54f-4591-b855-73014d18a06e-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-t7djv\" (UID: \"818838b2-f54f-4591-b855-73014d18a06e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t7djv" Oct 30 21:32:11.057573 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:32:11.057553 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/818838b2-f54f-4591-b855-73014d18a06e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-t7djv\" (UID: \"818838b2-f54f-4591-b855-73014d18a06e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t7djv" Oct 30 21:32:11.116437 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:32:11.116395 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/818838b2-f54f-4591-b855-73014d18a06e-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-t7djv\" (UID: \"818838b2-f54f-4591-b855-73014d18a06e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t7djv" Oct 30 21:32:11.116631 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:32:11.116606 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhw6f\" (UniqueName: \"kubernetes.io/projected/818838b2-f54f-4591-b855-73014d18a06e-kube-api-access-bhw6f\") pod \"maas-default-gateway-openshift-default-75d7c785f-t7djv\" (UID: \"818838b2-f54f-4591-b855-73014d18a06e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t7djv" Oct 30 21:32:11.118300 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:32:11.118283 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t7djv" Oct 30 21:32:11.547152 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:32:11.547125 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t7djv"] Oct 30 21:32:11.548585 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:32:11.548549 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod818838b2_f54f_4591_b855_73014d18a06e.slice/crio-ce17533b0c3bb8d210083d87ee4da97b7b99f915e8dd5ba8a3cd38bdfbcc2865 WatchSource:0}: Error finding container ce17533b0c3bb8d210083d87ee4da97b7b99f915e8dd5ba8a3cd38bdfbcc2865: Status 404 returned error can't find the container with id ce17533b0c3bb8d210083d87ee4da97b7b99f915e8dd5ba8a3cd38bdfbcc2865 Oct 30 21:32:11.792601 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:32:11.792562 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t7djv" event={"ID":"818838b2-f54f-4591-b855-73014d18a06e","Type":"ContainerStarted","Data":"ce17533b0c3bb8d210083d87ee4da97b7b99f915e8dd5ba8a3cd38bdfbcc2865"} Oct 30 21:32:15.789584 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:32:15.789551 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:32:15.789885 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:32:15.789622 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:32:15.789885 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:32:15.789655 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:32:16.828724 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:32:16.828682 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t7djv" event={"ID":"818838b2-f54f-4591-b855-73014d18a06e","Type":"ContainerStarted","Data":"8e8d2c844b854181452bd532cbedb6b871452df3fe11170cebeb2c75ee457588"} Oct 30 21:32:17.120829 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:32:17.120731 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t7djv" Oct 30 21:32:17.121235 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:32:17.121207 5748 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-t7djv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Oct 30 21:32:17.121421 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:32:17.121390 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t7djv" podUID="818838b2-f54f-4591-b855-73014d18a06e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Oct 30 21:32:18.120333 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:32:18.120293 5748 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-t7djv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Oct 30 21:32:18.120935 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:32:18.120883 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t7djv" podUID="818838b2-f54f-4591-b855-73014d18a06e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Oct 30 21:32:19.123385 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:32:19.123338 5748 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-t7djv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Oct 30 21:32:19.124073 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:32:19.124038 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t7djv" podUID="818838b2-f54f-4591-b855-73014d18a06e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Oct 30 21:32:20.122372 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:32:20.122330 5748 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-t7djv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Oct 30 21:32:20.122676 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:32:20.122640 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t7djv" podUID="818838b2-f54f-4591-b855-73014d18a06e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Oct 30 21:32:21.124922 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:32:21.122324 5748 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-t7djv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Oct 30 21:32:21.124922 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:32:21.122388 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t7djv" podUID="818838b2-f54f-4591-b855-73014d18a06e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Oct 30 21:32:22.119333 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:32:22.119292 5748 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-t7djv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Oct 30 21:32:22.119503 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:32:22.119360 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t7djv" podUID="818838b2-f54f-4591-b855-73014d18a06e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Oct 30 21:32:23.118755 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:32:23.118709 5748 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-t7djv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Oct 30 21:32:23.121105 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:32:23.118787 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t7djv" podUID="818838b2-f54f-4591-b855-73014d18a06e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Oct 30 21:32:24.123572 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:32:24.123536 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t7djv" Oct 30 21:32:24.124071 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:32:24.123836 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t7djv" Oct 30 21:32:24.124606 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:32:24.124585 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t7djv" Oct 30 21:32:24.296858 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:32:24.296801 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t7djv" podStartSLOduration=10.058533423 podStartE2EDuration="14.29678553s" totalImagesPullingTime="4.238252107s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:17.018411998 +0000 UTC m=+520.760932936" watchObservedRunningTime="2026-10-30 21:32:24.29678553 +0000 UTC m=+528.039306465" Oct 30 21:32:30.143983 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:32:30.143941 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:33:00.144822 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:33:00.144775 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:33:30.144607 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:33:30.144565 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:33:36.471882 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:33:36.471846 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j4v8r_3df036ff-1c00-4ce2-9e86-8acbeedf6c82/ovn-acl-logging/0.log" Oct 30 21:33:36.472667 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:33:36.472643 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j4v8r_3df036ff-1c00-4ce2-9e86-8acbeedf6c82/ovn-acl-logging/0.log" Oct 30 21:34:00.143587 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:34:00.143547 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:34:30.147186 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:34:30.147111 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:34:32.691005 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:32.690968 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-d68156-78b685b5dc-pwg7l"] Oct 30 21:34:32.694573 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:32.694553 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d68156-78b685b5dc-pwg7l" Oct 30 21:34:32.748236 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:32.748203 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-d68156\"" Oct 30 21:34:32.748236 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:32.748232 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-d68156-dockercfg-zmg6r\"" Oct 30 21:34:32.786931 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:32.786871 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqlnf\" (UniqueName: \"kubernetes.io/projected/8b269e36-1397-47d9-a349-04304ad783a4-kube-api-access-lqlnf\") pod \"payload-pre-processing-e2e-worker-w3-d68156-78b685b5dc-pwg7l\" (UID: \"8b269e36-1397-47d9-a349-04304ad783a4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d68156-78b685b5dc-pwg7l" Oct 30 21:34:32.787114 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:32.787034 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b269e36-1397-47d9-a349-04304ad783a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-d68156-78b685b5dc-pwg7l\" (UID: \"8b269e36-1397-47d9-a349-04304ad783a4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d68156-78b685b5dc-pwg7l" Oct 30 21:34:32.799874 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:32.799837 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-d68156-78b685b5dc-pwg7l"] Oct 30 21:34:32.887529 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:32.887482 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b269e36-1397-47d9-a349-04304ad783a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-d68156-78b685b5dc-pwg7l\" (UID: \"8b269e36-1397-47d9-a349-04304ad783a4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d68156-78b685b5dc-pwg7l" Oct 30 21:34:32.887729 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:32.887576 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqlnf\" (UniqueName: \"kubernetes.io/projected/8b269e36-1397-47d9-a349-04304ad783a4-kube-api-access-lqlnf\") pod \"payload-pre-processing-e2e-worker-w3-d68156-78b685b5dc-pwg7l\" (UID: \"8b269e36-1397-47d9-a349-04304ad783a4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d68156-78b685b5dc-pwg7l" Oct 30 21:34:32.888324 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:32.888304 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b269e36-1397-47d9-a349-04304ad783a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-d68156-78b685b5dc-pwg7l\" (UID: \"8b269e36-1397-47d9-a349-04304ad783a4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d68156-78b685b5dc-pwg7l" Oct 30 21:34:32.950312 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:32.950242 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqlnf\" (UniqueName: \"kubernetes.io/projected/8b269e36-1397-47d9-a349-04304ad783a4-kube-api-access-lqlnf\") pod \"payload-pre-processing-e2e-worker-w3-d68156-78b685b5dc-pwg7l\" (UID: \"8b269e36-1397-47d9-a349-04304ad783a4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d68156-78b685b5dc-pwg7l" Oct 30 21:34:33.008117 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:33.008083 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d68156-78b685b5dc-pwg7l" Oct 30 21:34:33.381699 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:33.381555 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-d68156-78b685b5dc-pwg7l"] Oct 30 21:34:33.384581 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:34:33.384543 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b269e36_1397_47d9_a349_04304ad783a4.slice/crio-469fbb88cb9f7ed76ce0ccb70520c01377000ae07791fdac3a3821158b8634a4 WatchSource:0}: Error finding container 469fbb88cb9f7ed76ce0ccb70520c01377000ae07791fdac3a3821158b8634a4: Status 404 returned error can't find the container with id 469fbb88cb9f7ed76ce0ccb70520c01377000ae07791fdac3a3821158b8634a4 Oct 30 21:34:33.386522 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:33.386499 5748 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:34:33.444362 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:33.444328 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d68156-78b685b5dc-pwg7l" event={"ID":"8b269e36-1397-47d9-a349-04304ad783a4","Type":"ContainerStarted","Data":"469fbb88cb9f7ed76ce0ccb70520c01377000ae07791fdac3a3821158b8634a4"} Oct 30 21:34:36.463056 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:36.463021 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d68156-78b685b5dc-pwg7l" event={"ID":"8b269e36-1397-47d9-a349-04304ad783a4","Type":"ContainerStarted","Data":"5560debf400a8ca000f0638c00abd098dd70e6616236a15d7727409a3fd3d78f"} Oct 30 21:34:36.463502 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:36.463152 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d68156-78b685b5dc-pwg7l" Oct 30 21:34:36.583696 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:36.583642 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d68156-78b685b5dc-pwg7l" podStartSLOduration=2.094649936 podStartE2EDuration="4.583629964s" totalImagesPullingTime="2.488980028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:34:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:34:36.583247996 +0000 UTC m=+660.325768930" watchObservedRunningTime="2026-10-30 21:34:36.583629964 +0000 UTC m=+660.326150898" Oct 30 21:34:42.434790 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:42.434753 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-69e4b2-77c69fd5cf-5jr47"] Oct 30 21:34:42.438505 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:42.438480 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-69e4b2-77c69fd5cf-5jr47" Oct 30 21:34:42.474505 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:42.474463 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-69e4b2-77c69fd5cf-5jr47"] Oct 30 21:34:42.474933 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:42.474909 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08366fbc-97d1-41a0-959a-447d00b7ec88-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-69e4b2-77c69fd5cf-5jr47\" (UID: \"08366fbc-97d1-41a0-959a-447d00b7ec88\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-69e4b2-77c69fd5cf-5jr47" Oct 30 21:34:42.475071 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:42.475045 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft9xz\" (UniqueName: \"kubernetes.io/projected/08366fbc-97d1-41a0-959a-447d00b7ec88-kube-api-access-ft9xz\") pod \"payload-pre-processing-e2e-worker-w6-69e4b2-77c69fd5cf-5jr47\" (UID: \"08366fbc-97d1-41a0-959a-447d00b7ec88\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-69e4b2-77c69fd5cf-5jr47" Oct 30 21:34:42.491871 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:42.491847 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-69e4b2\"" Oct 30 21:34:42.492358 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:42.492329 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-69e4b2-dockercfg-vpp22\"" Oct 30 21:34:42.575983 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:42.575942 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft9xz\" (UniqueName: \"kubernetes.io/projected/08366fbc-97d1-41a0-959a-447d00b7ec88-kube-api-access-ft9xz\") pod \"payload-pre-processing-e2e-worker-w6-69e4b2-77c69fd5cf-5jr47\" (UID: \"08366fbc-97d1-41a0-959a-447d00b7ec88\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-69e4b2-77c69fd5cf-5jr47" Oct 30 21:34:42.576184 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:42.576082 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08366fbc-97d1-41a0-959a-447d00b7ec88-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-69e4b2-77c69fd5cf-5jr47\" (UID: \"08366fbc-97d1-41a0-959a-447d00b7ec88\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-69e4b2-77c69fd5cf-5jr47" Oct 30 21:34:42.577124 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:42.577096 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08366fbc-97d1-41a0-959a-447d00b7ec88-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-69e4b2-77c69fd5cf-5jr47\" (UID: \"08366fbc-97d1-41a0-959a-447d00b7ec88\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-69e4b2-77c69fd5cf-5jr47" Oct 30 21:34:42.587188 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:42.587153 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-69e4b2-5845f89958-mzv58"] Oct 30 21:34:42.591574 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:42.591551 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-69e4b2-5845f89958-mzv58" Oct 30 21:34:42.638047 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:42.638017 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft9xz\" (UniqueName: \"kubernetes.io/projected/08366fbc-97d1-41a0-959a-447d00b7ec88-kube-api-access-ft9xz\") pod \"payload-pre-processing-e2e-worker-w6-69e4b2-77c69fd5cf-5jr47\" (UID: \"08366fbc-97d1-41a0-959a-447d00b7ec88\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-69e4b2-77c69fd5cf-5jr47" Oct 30 21:34:42.649371 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:42.649329 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-69e4b2-5845f89958-mzv58"] Oct 30 21:34:42.676767 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:42.676734 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ecabb08-132c-4b22-b550-77dbff23555d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-69e4b2-5845f89958-mzv58\" (UID: \"1ecabb08-132c-4b22-b550-77dbff23555d\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-69e4b2-5845f89958-mzv58" Oct 30 21:34:42.676947 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:42.676776 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnpmg\" (UniqueName: \"kubernetes.io/projected/1ecabb08-132c-4b22-b550-77dbff23555d-kube-api-access-rnpmg\") pod \"payload-processing-e2e-worker-w6-69e4b2-5845f89958-mzv58\" (UID: \"1ecabb08-132c-4b22-b550-77dbff23555d\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-69e4b2-5845f89958-mzv58" Oct 30 21:34:42.751472 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:42.751376 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-69e4b2-77c69fd5cf-5jr47" Oct 30 21:34:42.777855 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:42.777821 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ecabb08-132c-4b22-b550-77dbff23555d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-69e4b2-5845f89958-mzv58\" (UID: \"1ecabb08-132c-4b22-b550-77dbff23555d\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-69e4b2-5845f89958-mzv58" Oct 30 21:34:42.778424 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:42.777867 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnpmg\" (UniqueName: \"kubernetes.io/projected/1ecabb08-132c-4b22-b550-77dbff23555d-kube-api-access-rnpmg\") pod \"payload-processing-e2e-worker-w6-69e4b2-5845f89958-mzv58\" (UID: \"1ecabb08-132c-4b22-b550-77dbff23555d\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-69e4b2-5845f89958-mzv58" Oct 30 21:34:42.778653 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:42.778635 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ecabb08-132c-4b22-b550-77dbff23555d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-69e4b2-5845f89958-mzv58\" (UID: \"1ecabb08-132c-4b22-b550-77dbff23555d\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-69e4b2-5845f89958-mzv58" Oct 30 21:34:42.839765 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:42.839732 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnpmg\" (UniqueName: \"kubernetes.io/projected/1ecabb08-132c-4b22-b550-77dbff23555d-kube-api-access-rnpmg\") pod \"payload-processing-e2e-worker-w6-69e4b2-5845f89958-mzv58\" (UID: \"1ecabb08-132c-4b22-b550-77dbff23555d\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-69e4b2-5845f89958-mzv58" Oct 30 21:34:42.905071 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:42.905029 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-69e4b2-5845f89958-mzv58" Oct 30 21:34:43.126822 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:43.126793 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-69e4b2-77c69fd5cf-5jr47"] Oct 30 21:34:43.129088 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:34:43.129060 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08366fbc_97d1_41a0_959a_447d00b7ec88.slice/crio-5d02311587f0f72c092f74bbb60525b77945ec37872c669f0cba37de2fccf529 WatchSource:0}: Error finding container 5d02311587f0f72c092f74bbb60525b77945ec37872c669f0cba37de2fccf529: Status 404 returned error can't find the container with id 5d02311587f0f72c092f74bbb60525b77945ec37872c669f0cba37de2fccf529 Oct 30 21:34:43.292423 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:43.292281 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-69e4b2-5845f89958-mzv58"] Oct 30 21:34:43.295096 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:34:43.295069 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ecabb08_132c_4b22_b550_77dbff23555d.slice/crio-f97eededf4b894e1e40ff465f99b77a0253351cdb295955304149586765bbbe8 WatchSource:0}: Error finding container f97eededf4b894e1e40ff465f99b77a0253351cdb295955304149586765bbbe8: Status 404 returned error can't find the container with id f97eededf4b894e1e40ff465f99b77a0253351cdb295955304149586765bbbe8 Oct 30 21:34:43.495148 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:43.495105 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-69e4b2-5845f89958-mzv58" event={"ID":"1ecabb08-132c-4b22-b550-77dbff23555d","Type":"ContainerStarted","Data":"ef30e8cfa3a3c4963cebc0ebca7f1785ad17713ae417a3ba87292630051a036c"} Oct 30 21:34:43.495618 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:43.495160 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-69e4b2-5845f89958-mzv58" event={"ID":"1ecabb08-132c-4b22-b550-77dbff23555d","Type":"ContainerStarted","Data":"f97eededf4b894e1e40ff465f99b77a0253351cdb295955304149586765bbbe8"} Oct 30 21:34:43.495618 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:43.495220 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-69e4b2-5845f89958-mzv58" Oct 30 21:34:43.496716 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:43.496692 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-69e4b2-77c69fd5cf-5jr47" event={"ID":"08366fbc-97d1-41a0-959a-447d00b7ec88","Type":"ContainerStarted","Data":"f69d487bf6d71e9461558b17030c95f7755742631a35751108b038a6aa1c5f76"} Oct 30 21:34:43.496841 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:43.496727 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-69e4b2-77c69fd5cf-5jr47" event={"ID":"08366fbc-97d1-41a0-959a-447d00b7ec88","Type":"ContainerStarted","Data":"5d02311587f0f72c092f74bbb60525b77945ec37872c669f0cba37de2fccf529"} Oct 30 21:34:43.496841 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:43.496827 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-69e4b2-77c69fd5cf-5jr47" Oct 30 21:34:43.613166 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:43.613113 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-69e4b2-5845f89958-mzv58" podStartSLOduration=1.613099463 podStartE2EDuration="1.613099463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:34:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:34:43.612817518 +0000 UTC m=+667.355338468" watchObservedRunningTime="2026-10-30 21:34:43.613099463 +0000 UTC m=+667.355620397" Oct 30 21:34:43.731642 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:43.731587 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-69e4b2-77c69fd5cf-5jr47" podStartSLOduration=1.731572612 podStartE2EDuration="1.731572612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:34:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:34:43.729992 +0000 UTC m=+667.472512932" watchObservedRunningTime="2026-10-30 21:34:43.731572612 +0000 UTC m=+667.474093546" Oct 30 21:34:44.949239 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:44.949203 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-691598-85b4f65bbc-k4gtg"] Oct 30 21:34:44.952758 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:44.952733 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-691598-85b4f65bbc-k4gtg" Oct 30 21:34:44.998753 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:44.998714 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvrt5\" (UniqueName: \"kubernetes.io/projected/6c34df93-2d70-41c6-9e31-8b7e30fdbe4c-kube-api-access-lvrt5\") pod \"payload-pre-processing-e2e-worker-w0-691598-85b4f65bbc-k4gtg\" (UID: \"6c34df93-2d70-41c6-9e31-8b7e30fdbe4c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-691598-85b4f65bbc-k4gtg" Oct 30 21:34:44.998955 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:44.998852 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c34df93-2d70-41c6-9e31-8b7e30fdbe4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-691598-85b4f65bbc-k4gtg\" (UID: \"6c34df93-2d70-41c6-9e31-8b7e30fdbe4c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-691598-85b4f65bbc-k4gtg" Oct 30 21:34:45.006003 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:45.005969 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-691598\"" Oct 30 21:34:45.056320 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:45.056292 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-691598-dockercfg-w5z99\"" Oct 30 21:34:45.058408 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:45.058379 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-691598-85b4f65bbc-k4gtg"] Oct 30 21:34:45.099342 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:45.099307 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c34df93-2d70-41c6-9e31-8b7e30fdbe4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-691598-85b4f65bbc-k4gtg\" (UID: \"6c34df93-2d70-41c6-9e31-8b7e30fdbe4c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-691598-85b4f65bbc-k4gtg" Oct 30 21:34:45.099531 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:45.099368 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvrt5\" (UniqueName: \"kubernetes.io/projected/6c34df93-2d70-41c6-9e31-8b7e30fdbe4c-kube-api-access-lvrt5\") pod \"payload-pre-processing-e2e-worker-w0-691598-85b4f65bbc-k4gtg\" (UID: \"6c34df93-2d70-41c6-9e31-8b7e30fdbe4c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-691598-85b4f65bbc-k4gtg" Oct 30 21:34:45.100156 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:45.100136 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c34df93-2d70-41c6-9e31-8b7e30fdbe4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-691598-85b4f65bbc-k4gtg\" (UID: \"6c34df93-2d70-41c6-9e31-8b7e30fdbe4c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-691598-85b4f65bbc-k4gtg" Oct 30 21:34:45.108184 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:45.108150 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-691598-f774fbb9d-6fxfj"] Oct 30 21:34:45.111667 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:45.111645 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-691598-f774fbb9d-6fxfj" Oct 30 21:34:45.156046 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:45.156013 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-691598-f774fbb9d-6fxfj"] Oct 30 21:34:45.162540 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:45.162508 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvrt5\" (UniqueName: \"kubernetes.io/projected/6c34df93-2d70-41c6-9e31-8b7e30fdbe4c-kube-api-access-lvrt5\") pod \"payload-pre-processing-e2e-worker-w0-691598-85b4f65bbc-k4gtg\" (UID: \"6c34df93-2d70-41c6-9e31-8b7e30fdbe4c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-691598-85b4f65bbc-k4gtg" Oct 30 21:34:45.200701 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:45.200612 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ce6a432-f03c-4e18-abe1-5d5da48d092d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-691598-f774fbb9d-6fxfj\" (UID: \"3ce6a432-f03c-4e18-abe1-5d5da48d092d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-691598-f774fbb9d-6fxfj" Oct 30 21:34:45.200701 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:45.200649 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7cjk\" (UniqueName: \"kubernetes.io/projected/3ce6a432-f03c-4e18-abe1-5d5da48d092d-kube-api-access-j7cjk\") pod \"payload-processing-e2e-worker-w0-691598-f774fbb9d-6fxfj\" (UID: \"3ce6a432-f03c-4e18-abe1-5d5da48d092d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-691598-f774fbb9d-6fxfj" Oct 30 21:34:45.266124 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:45.266090 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-691598-85b4f65bbc-k4gtg" Oct 30 21:34:45.302234 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:45.302198 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ce6a432-f03c-4e18-abe1-5d5da48d092d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-691598-f774fbb9d-6fxfj\" (UID: \"3ce6a432-f03c-4e18-abe1-5d5da48d092d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-691598-f774fbb9d-6fxfj" Oct 30 21:34:45.302234 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:45.302237 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7cjk\" (UniqueName: \"kubernetes.io/projected/3ce6a432-f03c-4e18-abe1-5d5da48d092d-kube-api-access-j7cjk\") pod \"payload-processing-e2e-worker-w0-691598-f774fbb9d-6fxfj\" (UID: \"3ce6a432-f03c-4e18-abe1-5d5da48d092d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-691598-f774fbb9d-6fxfj" Oct 30 21:34:45.302981 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:45.302956 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ce6a432-f03c-4e18-abe1-5d5da48d092d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-691598-f774fbb9d-6fxfj\" (UID: \"3ce6a432-f03c-4e18-abe1-5d5da48d092d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-691598-f774fbb9d-6fxfj" Oct 30 21:34:45.364505 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:45.364466 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7cjk\" (UniqueName: \"kubernetes.io/projected/3ce6a432-f03c-4e18-abe1-5d5da48d092d-kube-api-access-j7cjk\") pod \"payload-processing-e2e-worker-w0-691598-f774fbb9d-6fxfj\" (UID: \"3ce6a432-f03c-4e18-abe1-5d5da48d092d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-691598-f774fbb9d-6fxfj" Oct 30 21:34:45.424383 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:45.424339 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-691598-f774fbb9d-6fxfj" Oct 30 21:34:45.638535 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:45.638492 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-691598-85b4f65bbc-k4gtg"] Oct 30 21:34:45.641226 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:34:45.641199 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c34df93_2d70_41c6_9e31_8b7e30fdbe4c.slice/crio-ca452dc71fe19c8796cf7258b79c0fbe4bdbeca8416fbf20b1522d3f0e99a770 WatchSource:0}: Error finding container ca452dc71fe19c8796cf7258b79c0fbe4bdbeca8416fbf20b1522d3f0e99a770: Status 404 returned error can't find the container with id ca452dc71fe19c8796cf7258b79c0fbe4bdbeca8416fbf20b1522d3f0e99a770 Oct 30 21:34:45.797552 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:45.797475 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-691598-f774fbb9d-6fxfj"] Oct 30 21:34:45.800697 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:34:45.800666 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ce6a432_f03c_4e18_abe1_5d5da48d092d.slice/crio-388e86abd9f680fa372f5c6368a0208578baab15c09f544758818caf0487bd59 WatchSource:0}: Error finding container 388e86abd9f680fa372f5c6368a0208578baab15c09f544758818caf0487bd59: Status 404 returned error can't find the container with id 388e86abd9f680fa372f5c6368a0208578baab15c09f544758818caf0487bd59 Oct 30 21:34:46.512155 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:46.512118 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-691598-85b4f65bbc-k4gtg" event={"ID":"6c34df93-2d70-41c6-9e31-8b7e30fdbe4c","Type":"ContainerStarted","Data":"ec57c92c6963ec711cf9824946ced346e1f27d16ce78f699d24221ddfa39dfb7"} Oct 30 21:34:46.512614 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:46.512178 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-691598-85b4f65bbc-k4gtg" event={"ID":"6c34df93-2d70-41c6-9e31-8b7e30fdbe4c","Type":"ContainerStarted","Data":"ca452dc71fe19c8796cf7258b79c0fbe4bdbeca8416fbf20b1522d3f0e99a770"} Oct 30 21:34:46.512614 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:46.512302 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-691598-85b4f65bbc-k4gtg" Oct 30 21:34:46.514062 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:46.514035 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-691598-f774fbb9d-6fxfj" event={"ID":"3ce6a432-f03c-4e18-abe1-5d5da48d092d","Type":"ContainerStarted","Data":"4f867897fe6047f26cb8c3818dfa4d6f47b0a09ea80ac34b243375dd89b00c4e"} Oct 30 21:34:46.514187 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:46.514093 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-691598-f774fbb9d-6fxfj" Oct 30 21:34:46.514187 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:46.514112 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-691598-f774fbb9d-6fxfj" event={"ID":"3ce6a432-f03c-4e18-abe1-5d5da48d092d","Type":"ContainerStarted","Data":"388e86abd9f680fa372f5c6368a0208578baab15c09f544758818caf0487bd59"} Oct 30 21:34:46.629589 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:46.629527 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-691598-85b4f65bbc-k4gtg" podStartSLOduration=2.629511596 podStartE2EDuration="2.629511596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:34:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:34:46.628502037 +0000 UTC m=+670.371022973" watchObservedRunningTime="2026-10-30 21:34:46.629511596 +0000 UTC m=+670.372032531" Oct 30 21:34:46.746825 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:46.746766 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-691598-f774fbb9d-6fxfj" podStartSLOduration=1.746753397 podStartE2EDuration="1.746753397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:34:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:34:46.745662286 +0000 UTC m=+670.488183223" watchObservedRunningTime="2026-10-30 21:34:46.746753397 +0000 UTC m=+670.489274310" Oct 30 21:34:47.471221 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:47.471191 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d68156-78b685b5dc-pwg7l" Oct 30 21:34:54.504615 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:54.504586 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-69e4b2-5845f89958-mzv58" Oct 30 21:34:54.505039 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:54.504652 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-69e4b2-77c69fd5cf-5jr47" Oct 30 21:34:54.669519 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:54.669443 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-6ea374-df875f76c-8bbkw"] Oct 30 21:34:54.673995 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:54.673956 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6ea374-df875f76c-8bbkw" Oct 30 21:34:54.727203 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:54.727168 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-6ea374\"" Oct 30 21:34:54.727345 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:54.727232 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-6ea374-dockercfg-f9nzc\"" Oct 30 21:34:54.738867 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:54.738830 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-6ea374-df875f76c-8bbkw"] Oct 30 21:34:54.794631 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:54.794591 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/084e23c1-9a28-4b27-ab32-d3c348fd2c1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-6ea374-df875f76c-8bbkw\" (UID: \"084e23c1-9a28-4b27-ab32-d3c348fd2c1b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6ea374-df875f76c-8bbkw" Oct 30 21:34:54.794841 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:54.794732 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvfh8\" (UniqueName: \"kubernetes.io/projected/084e23c1-9a28-4b27-ab32-d3c348fd2c1b-kube-api-access-jvfh8\") pod \"payload-pre-processing-e2e-worker-w1-6ea374-df875f76c-8bbkw\" (UID: \"084e23c1-9a28-4b27-ab32-d3c348fd2c1b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6ea374-df875f76c-8bbkw" Oct 30 21:34:54.803147 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:54.803117 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-6ea374-84496c9544-jjw75"] Oct 30 21:34:54.809643 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:54.809621 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-6ea374-84496c9544-jjw75" Oct 30 21:34:54.866088 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:54.865650 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-6ea374-84496c9544-jjw75"] Oct 30 21:34:54.895366 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:54.895327 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eed8b35c-e416-4173-98a7-ad7b3d6bc0ea-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-6ea374-84496c9544-jjw75\" (UID: \"eed8b35c-e416-4173-98a7-ad7b3d6bc0ea\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-6ea374-84496c9544-jjw75" Oct 30 21:34:54.895366 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:54.895367 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8mxn\" (UniqueName: \"kubernetes.io/projected/eed8b35c-e416-4173-98a7-ad7b3d6bc0ea-kube-api-access-f8mxn\") pod \"payload-processing-e2e-worker-w1-6ea374-84496c9544-jjw75\" (UID: \"eed8b35c-e416-4173-98a7-ad7b3d6bc0ea\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-6ea374-84496c9544-jjw75" Oct 30 21:34:54.895596 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:54.895391 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/084e23c1-9a28-4b27-ab32-d3c348fd2c1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-6ea374-df875f76c-8bbkw\" (UID: \"084e23c1-9a28-4b27-ab32-d3c348fd2c1b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6ea374-df875f76c-8bbkw" Oct 30 21:34:54.895596 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:54.895576 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvfh8\" (UniqueName: \"kubernetes.io/projected/084e23c1-9a28-4b27-ab32-d3c348fd2c1b-kube-api-access-jvfh8\") pod \"payload-pre-processing-e2e-worker-w1-6ea374-df875f76c-8bbkw\" (UID: \"084e23c1-9a28-4b27-ab32-d3c348fd2c1b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6ea374-df875f76c-8bbkw" Oct 30 21:34:54.896364 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:54.896330 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/084e23c1-9a28-4b27-ab32-d3c348fd2c1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-6ea374-df875f76c-8bbkw\" (UID: \"084e23c1-9a28-4b27-ab32-d3c348fd2c1b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6ea374-df875f76c-8bbkw" Oct 30 21:34:54.957993 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:54.957911 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvfh8\" (UniqueName: \"kubernetes.io/projected/084e23c1-9a28-4b27-ab32-d3c348fd2c1b-kube-api-access-jvfh8\") pod \"payload-pre-processing-e2e-worker-w1-6ea374-df875f76c-8bbkw\" (UID: \"084e23c1-9a28-4b27-ab32-d3c348fd2c1b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6ea374-df875f76c-8bbkw" Oct 30 21:34:54.988554 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:54.988523 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6ea374-df875f76c-8bbkw" Oct 30 21:34:54.997034 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:54.997012 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eed8b35c-e416-4173-98a7-ad7b3d6bc0ea-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-6ea374-84496c9544-jjw75\" (UID: \"eed8b35c-e416-4173-98a7-ad7b3d6bc0ea\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-6ea374-84496c9544-jjw75" Oct 30 21:34:54.997129 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:54.997051 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8mxn\" (UniqueName: \"kubernetes.io/projected/eed8b35c-e416-4173-98a7-ad7b3d6bc0ea-kube-api-access-f8mxn\") pod \"payload-processing-e2e-worker-w1-6ea374-84496c9544-jjw75\" (UID: \"eed8b35c-e416-4173-98a7-ad7b3d6bc0ea\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-6ea374-84496c9544-jjw75" Oct 30 21:34:55.001029 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:55.000882 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eed8b35c-e416-4173-98a7-ad7b3d6bc0ea-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-6ea374-84496c9544-jjw75\" (UID: \"eed8b35c-e416-4173-98a7-ad7b3d6bc0ea\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-6ea374-84496c9544-jjw75" Oct 30 21:34:55.059352 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:55.059318 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8mxn\" (UniqueName: \"kubernetes.io/projected/eed8b35c-e416-4173-98a7-ad7b3d6bc0ea-kube-api-access-f8mxn\") pod \"payload-processing-e2e-worker-w1-6ea374-84496c9544-jjw75\" (UID: \"eed8b35c-e416-4173-98a7-ad7b3d6bc0ea\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-6ea374-84496c9544-jjw75" Oct 30 21:34:55.122545 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:55.122509 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-6ea374-84496c9544-jjw75" Oct 30 21:34:55.363581 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:55.363398 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-6ea374-df875f76c-8bbkw"] Oct 30 21:34:55.365749 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:34:55.365714 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod084e23c1_9a28_4b27_ab32_d3c348fd2c1b.slice/crio-56bb3285b757dba2b4fef354747ace90d2bedd91dd51743ef9b6164c1282e3a3 WatchSource:0}: Error finding container 56bb3285b757dba2b4fef354747ace90d2bedd91dd51743ef9b6164c1282e3a3: Status 404 returned error can't find the container with id 56bb3285b757dba2b4fef354747ace90d2bedd91dd51743ef9b6164c1282e3a3 Oct 30 21:34:55.494935 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:55.494882 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-6ea374-84496c9544-jjw75"] Oct 30 21:34:55.498613 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:34:55.498584 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeed8b35c_e416_4173_98a7_ad7b3d6bc0ea.slice/crio-65e300893b541cd56bc5fa57fd193eeb304bc7134de8a01ae811c8e317cc4d88 WatchSource:0}: Error finding container 65e300893b541cd56bc5fa57fd193eeb304bc7134de8a01ae811c8e317cc4d88: Status 404 returned error can't find the container with id 65e300893b541cd56bc5fa57fd193eeb304bc7134de8a01ae811c8e317cc4d88 Oct 30 21:34:55.559757 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:55.559726 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-6ea374-84496c9544-jjw75" event={"ID":"eed8b35c-e416-4173-98a7-ad7b3d6bc0ea","Type":"ContainerStarted","Data":"53f3a622dd741bc5719560eedbef8f8c45e92a74d6de112144dd104e7270ee6a"} Oct 30 21:34:55.560203 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:55.559782 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-6ea374-84496c9544-jjw75" event={"ID":"eed8b35c-e416-4173-98a7-ad7b3d6bc0ea","Type":"ContainerStarted","Data":"65e300893b541cd56bc5fa57fd193eeb304bc7134de8a01ae811c8e317cc4d88"} Oct 30 21:34:55.560203 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:55.559873 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-6ea374-84496c9544-jjw75" Oct 30 21:34:55.561569 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:55.561545 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6ea374-df875f76c-8bbkw" event={"ID":"084e23c1-9a28-4b27-ab32-d3c348fd2c1b","Type":"ContainerStarted","Data":"0d2b4c6501d7111d9d03377c4aca0ace6a4ba091efd1ca3b803b5155e4e96c43"} Oct 30 21:34:55.561663 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:55.561586 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6ea374-df875f76c-8bbkw" event={"ID":"084e23c1-9a28-4b27-ab32-d3c348fd2c1b","Type":"ContainerStarted","Data":"56bb3285b757dba2b4fef354747ace90d2bedd91dd51743ef9b6164c1282e3a3"} Oct 30 21:34:55.561663 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:55.561613 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6ea374-df875f76c-8bbkw" Oct 30 21:34:55.679378 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:55.679322 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-6ea374-84496c9544-jjw75" podStartSLOduration=1.679310336 podStartE2EDuration="1.679310336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:34:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:34:55.678447392 +0000 UTC m=+679.420968327" watchObservedRunningTime="2026-10-30 21:34:55.679310336 +0000 UTC m=+679.421831308" Oct 30 21:34:55.796779 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:55.796723 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6ea374-df875f76c-8bbkw" podStartSLOduration=1.796708615 podStartE2EDuration="1.796708615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:34:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:34:55.795593851 +0000 UTC m=+679.538114786" watchObservedRunningTime="2026-10-30 21:34:55.796708615 +0000 UTC m=+679.539229550" Oct 30 21:34:57.522651 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:57.522622 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-691598-f774fbb9d-6fxfj" Oct 30 21:34:57.523092 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:34:57.522685 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-691598-85b4f65bbc-k4gtg" Oct 30 21:35:00.142841 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:35:00.142809 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:35:06.568892 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:06.568862 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-6ea374-84496c9544-jjw75" Oct 30 21:35:06.569433 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:06.568961 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6ea374-df875f76c-8bbkw" Oct 30 21:35:08.817586 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:08.817541 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-d68156/simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk"] Oct 30 21:35:08.822010 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:08.821984 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d68156/simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk" Oct 30 21:35:08.831615 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:08.831593 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db285e9c-ca87-43fa-8f06-69e271e5c2c5-tls-certs\") pod \"simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk\" (UID: \"db285e9c-ca87-43fa-8f06-69e271e5c2c5\") " pod="e2e-models-e2e-worker-w3-d68156/simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk" Oct 30 21:35:08.831729 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:08.831622 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db285e9c-ca87-43fa-8f06-69e271e5c2c5-home\") pod \"simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk\" (UID: \"db285e9c-ca87-43fa-8f06-69e271e5c2c5\") " pod="e2e-models-e2e-worker-w3-d68156/simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk" Oct 30 21:35:08.831729 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:08.831641 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db285e9c-ca87-43fa-8f06-69e271e5c2c5-tmp-dir\") pod \"simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk\" (UID: \"db285e9c-ca87-43fa-8f06-69e271e5c2c5\") " pod="e2e-models-e2e-worker-w3-d68156/simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk" Oct 30 21:35:08.831796 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:08.831743 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db285e9c-ca87-43fa-8f06-69e271e5c2c5-model-cache\") pod \"simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk\" (UID: \"db285e9c-ca87-43fa-8f06-69e271e5c2c5\") " pod="e2e-models-e2e-worker-w3-d68156/simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk" Oct 30 21:35:08.831796 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:08.831785 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db285e9c-ca87-43fa-8f06-69e271e5c2c5-dshm\") pod \"simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk\" (UID: \"db285e9c-ca87-43fa-8f06-69e271e5c2c5\") " pod="e2e-models-e2e-worker-w3-d68156/simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk" Oct 30 21:35:08.831858 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:08.831802 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvwll\" (UniqueName: \"kubernetes.io/projected/db285e9c-ca87-43fa-8f06-69e271e5c2c5-kube-api-access-cvwll\") pod \"simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk\" (UID: \"db285e9c-ca87-43fa-8f06-69e271e5c2c5\") " pod="e2e-models-e2e-worker-w3-d68156/simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk" Oct 30 21:35:08.933044 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:08.933007 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db285e9c-ca87-43fa-8f06-69e271e5c2c5-dshm\") pod \"simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk\" (UID: \"db285e9c-ca87-43fa-8f06-69e271e5c2c5\") " pod="e2e-models-e2e-worker-w3-d68156/simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk" Oct 30 21:35:08.933044 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:08.933051 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvwll\" (UniqueName: \"kubernetes.io/projected/db285e9c-ca87-43fa-8f06-69e271e5c2c5-kube-api-access-cvwll\") pod \"simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk\" (UID: \"db285e9c-ca87-43fa-8f06-69e271e5c2c5\") " pod="e2e-models-e2e-worker-w3-d68156/simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk" Oct 30 21:35:08.933311 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:08.933150 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db285e9c-ca87-43fa-8f06-69e271e5c2c5-tls-certs\") pod \"simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk\" (UID: \"db285e9c-ca87-43fa-8f06-69e271e5c2c5\") " pod="e2e-models-e2e-worker-w3-d68156/simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk" Oct 30 21:35:08.933311 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:08.933181 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db285e9c-ca87-43fa-8f06-69e271e5c2c5-home\") pod \"simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk\" (UID: \"db285e9c-ca87-43fa-8f06-69e271e5c2c5\") " pod="e2e-models-e2e-worker-w3-d68156/simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk" Oct 30 21:35:08.933311 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:08.933203 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db285e9c-ca87-43fa-8f06-69e271e5c2c5-tmp-dir\") pod \"simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk\" (UID: \"db285e9c-ca87-43fa-8f06-69e271e5c2c5\") " pod="e2e-models-e2e-worker-w3-d68156/simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk" Oct 30 21:35:08.933311 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:08.933248 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db285e9c-ca87-43fa-8f06-69e271e5c2c5-model-cache\") pod \"simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk\" (UID: \"db285e9c-ca87-43fa-8f06-69e271e5c2c5\") " pod="e2e-models-e2e-worker-w3-d68156/simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk" Oct 30 21:35:08.933607 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:08.933584 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db285e9c-ca87-43fa-8f06-69e271e5c2c5-home\") pod \"simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk\" (UID: \"db285e9c-ca87-43fa-8f06-69e271e5c2c5\") " pod="e2e-models-e2e-worker-w3-d68156/simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk" Oct 30 21:35:08.933651 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:08.933601 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db285e9c-ca87-43fa-8f06-69e271e5c2c5-tmp-dir\") pod \"simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk\" (UID: \"db285e9c-ca87-43fa-8f06-69e271e5c2c5\") " pod="e2e-models-e2e-worker-w3-d68156/simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk" Oct 30 21:35:08.933651 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:08.933627 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db285e9c-ca87-43fa-8f06-69e271e5c2c5-model-cache\") pod \"simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk\" (UID: \"db285e9c-ca87-43fa-8f06-69e271e5c2c5\") " pod="e2e-models-e2e-worker-w3-d68156/simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk" Oct 30 21:35:08.935291 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:08.935270 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db285e9c-ca87-43fa-8f06-69e271e5c2c5-dshm\") pod \"simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk\" (UID: \"db285e9c-ca87-43fa-8f06-69e271e5c2c5\") " pod="e2e-models-e2e-worker-w3-d68156/simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk" Oct 30 21:35:08.942464 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:08.942430 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-d68156\"/\"simulator-w3-d68156-kserve-self-signed-certs\"" Oct 30 21:35:08.942464 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:08.942430 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-d68156\"/\"openshift-service-ca.crt\"" Oct 30 21:35:08.942647 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:08.942599 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-d68156\"/\"default-dockercfg-h647m\"" Oct 30 21:35:08.944392 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:08.944367 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d68156/simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk"] Oct 30 21:35:08.946058 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:08.946038 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db285e9c-ca87-43fa-8f06-69e271e5c2c5-tls-certs\") pod \"simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk\" (UID: \"db285e9c-ca87-43fa-8f06-69e271e5c2c5\") " pod="e2e-models-e2e-worker-w3-d68156/simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk" Oct 30 21:35:08.952347 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:08.952329 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-d68156\"/\"kube-root-ca.crt\"" Oct 30 21:35:09.005762 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:09.005730 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvwll\" (UniqueName: \"kubernetes.io/projected/db285e9c-ca87-43fa-8f06-69e271e5c2c5-kube-api-access-cvwll\") pod \"simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk\" (UID: \"db285e9c-ca87-43fa-8f06-69e271e5c2c5\") " pod="e2e-models-e2e-worker-w3-d68156/simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk" Oct 30 21:35:09.134676 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:09.134587 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d68156/simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk" Oct 30 21:35:09.525158 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:09.525127 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d68156/simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk"] Oct 30 21:35:09.528708 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:35:09.528678 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddb285e9c_ca87_43fa_8f06_69e271e5c2c5.slice/crio-cac752eb6b66afdf26b01a667ea0e6c22419b0a3eea717afe0ddf02a971c7cc0 WatchSource:0}: Error finding container cac752eb6b66afdf26b01a667ea0e6c22419b0a3eea717afe0ddf02a971c7cc0: Status 404 returned error can't find the container with id cac752eb6b66afdf26b01a667ea0e6c22419b0a3eea717afe0ddf02a971c7cc0 Oct 30 21:35:09.632562 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:09.632525 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d68156/simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk" event={"ID":"db285e9c-ca87-43fa-8f06-69e271e5c2c5","Type":"ContainerStarted","Data":"cac752eb6b66afdf26b01a667ea0e6c22419b0a3eea717afe0ddf02a971c7cc0"} Oct 30 21:35:12.650987 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:12.650951 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d68156/simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk" event={"ID":"db285e9c-ca87-43fa-8f06-69e271e5c2c5","Type":"ContainerStarted","Data":"648bf1c437fe90e4c4cbe008c5c76fc9c55248af538d84a3c0a526c84f5512e3"} Oct 30 21:35:12.770230 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:12.770180 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-d68156/simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk" podStartSLOduration=2.722295489 podStartE2EDuration="4.770165656s" totalImagesPullingTime="2.047870167s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:12.768721576 +0000 UTC m=+696.511242512" watchObservedRunningTime="2026-10-30 21:35:12.770165656 +0000 UTC m=+696.512686621" Oct 30 21:35:13.135728 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:13.135647 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d68156/simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk" Oct 30 21:35:19.135230 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:19.135183 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-d68156/simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk" Oct 30 21:35:19.148044 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:19.148017 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-d68156/simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk" Oct 30 21:35:19.701136 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:19.701103 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d68156/simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk" Oct 30 21:35:30.147851 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:35:30.147812 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:35:31.723763 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:31.723720 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-1de75a/simulator-w4-1de75a-kserve-6887dd4b46-kc5c2"] Oct 30 21:35:31.729046 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:31.729022 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1de75a/simulator-w4-1de75a-kserve-6887dd4b46-kc5c2" Oct 30 21:35:31.835697 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:31.835660 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-1de75a\"/\"openshift-service-ca.crt\"" Oct 30 21:35:31.836038 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:31.835996 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-1de75a\"/\"kube-root-ca.crt\"" Oct 30 21:35:31.836101 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:31.836074 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-1de75a\"/\"simulator-w4-1de75a-kserve-self-signed-certs\"" Oct 30 21:35:31.836258 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:31.836240 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-1de75a\"/\"default-dockercfg-tv786\"" Oct 30 21:35:31.837423 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:31.837403 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1de75a/simulator-w4-1de75a-kserve-6887dd4b46-kc5c2"] Oct 30 21:35:31.857201 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:31.857170 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/926fa847-0c3c-43e0-b378-04c82433cd98-model-cache\") pod \"simulator-w4-1de75a-kserve-6887dd4b46-kc5c2\" (UID: \"926fa847-0c3c-43e0-b378-04c82433cd98\") " pod="e2e-models-e2e-worker-w4-1de75a/simulator-w4-1de75a-kserve-6887dd4b46-kc5c2" Oct 30 21:35:31.857336 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:31.857228 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/926fa847-0c3c-43e0-b378-04c82433cd98-home\") pod \"simulator-w4-1de75a-kserve-6887dd4b46-kc5c2\" (UID: \"926fa847-0c3c-43e0-b378-04c82433cd98\") " pod="e2e-models-e2e-worker-w4-1de75a/simulator-w4-1de75a-kserve-6887dd4b46-kc5c2" Oct 30 21:35:31.857336 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:31.857294 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfqvl\" (UniqueName: \"kubernetes.io/projected/926fa847-0c3c-43e0-b378-04c82433cd98-kube-api-access-kfqvl\") pod \"simulator-w4-1de75a-kserve-6887dd4b46-kc5c2\" (UID: \"926fa847-0c3c-43e0-b378-04c82433cd98\") " pod="e2e-models-e2e-worker-w4-1de75a/simulator-w4-1de75a-kserve-6887dd4b46-kc5c2" Oct 30 21:35:31.857336 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:31.857319 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/926fa847-0c3c-43e0-b378-04c82433cd98-dshm\") pod \"simulator-w4-1de75a-kserve-6887dd4b46-kc5c2\" (UID: \"926fa847-0c3c-43e0-b378-04c82433cd98\") " pod="e2e-models-e2e-worker-w4-1de75a/simulator-w4-1de75a-kserve-6887dd4b46-kc5c2" Oct 30 21:35:31.857464 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:31.857417 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/926fa847-0c3c-43e0-b378-04c82433cd98-tls-certs\") pod \"simulator-w4-1de75a-kserve-6887dd4b46-kc5c2\" (UID: \"926fa847-0c3c-43e0-b378-04c82433cd98\") " pod="e2e-models-e2e-worker-w4-1de75a/simulator-w4-1de75a-kserve-6887dd4b46-kc5c2" Oct 30 21:35:31.857464 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:31.857452 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/926fa847-0c3c-43e0-b378-04c82433cd98-tmp-dir\") pod \"simulator-w4-1de75a-kserve-6887dd4b46-kc5c2\" (UID: \"926fa847-0c3c-43e0-b378-04c82433cd98\") " pod="e2e-models-e2e-worker-w4-1de75a/simulator-w4-1de75a-kserve-6887dd4b46-kc5c2" Oct 30 21:35:31.958930 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:31.958872 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/926fa847-0c3c-43e0-b378-04c82433cd98-tls-certs\") pod \"simulator-w4-1de75a-kserve-6887dd4b46-kc5c2\" (UID: \"926fa847-0c3c-43e0-b378-04c82433cd98\") " pod="e2e-models-e2e-worker-w4-1de75a/simulator-w4-1de75a-kserve-6887dd4b46-kc5c2" Oct 30 21:35:31.958930 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:31.958937 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/926fa847-0c3c-43e0-b378-04c82433cd98-tmp-dir\") pod \"simulator-w4-1de75a-kserve-6887dd4b46-kc5c2\" (UID: \"926fa847-0c3c-43e0-b378-04c82433cd98\") " pod="e2e-models-e2e-worker-w4-1de75a/simulator-w4-1de75a-kserve-6887dd4b46-kc5c2" Oct 30 21:35:31.959196 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:31.958956 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/926fa847-0c3c-43e0-b378-04c82433cd98-model-cache\") pod \"simulator-w4-1de75a-kserve-6887dd4b46-kc5c2\" (UID: \"926fa847-0c3c-43e0-b378-04c82433cd98\") " pod="e2e-models-e2e-worker-w4-1de75a/simulator-w4-1de75a-kserve-6887dd4b46-kc5c2" Oct 30 21:35:31.959196 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:31.959000 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/926fa847-0c3c-43e0-b378-04c82433cd98-home\") pod \"simulator-w4-1de75a-kserve-6887dd4b46-kc5c2\" (UID: \"926fa847-0c3c-43e0-b378-04c82433cd98\") " pod="e2e-models-e2e-worker-w4-1de75a/simulator-w4-1de75a-kserve-6887dd4b46-kc5c2" Oct 30 21:35:31.959196 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:31.959023 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfqvl\" (UniqueName: \"kubernetes.io/projected/926fa847-0c3c-43e0-b378-04c82433cd98-kube-api-access-kfqvl\") pod \"simulator-w4-1de75a-kserve-6887dd4b46-kc5c2\" (UID: \"926fa847-0c3c-43e0-b378-04c82433cd98\") " pod="e2e-models-e2e-worker-w4-1de75a/simulator-w4-1de75a-kserve-6887dd4b46-kc5c2" Oct 30 21:35:31.959196 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:31.959051 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/926fa847-0c3c-43e0-b378-04c82433cd98-dshm\") pod \"simulator-w4-1de75a-kserve-6887dd4b46-kc5c2\" (UID: \"926fa847-0c3c-43e0-b378-04c82433cd98\") " pod="e2e-models-e2e-worker-w4-1de75a/simulator-w4-1de75a-kserve-6887dd4b46-kc5c2" Oct 30 21:35:31.959547 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:31.959522 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/926fa847-0c3c-43e0-b378-04c82433cd98-tmp-dir\") pod \"simulator-w4-1de75a-kserve-6887dd4b46-kc5c2\" (UID: \"926fa847-0c3c-43e0-b378-04c82433cd98\") " pod="e2e-models-e2e-worker-w4-1de75a/simulator-w4-1de75a-kserve-6887dd4b46-kc5c2" Oct 30 21:35:31.959547 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:31.959540 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/926fa847-0c3c-43e0-b378-04c82433cd98-home\") pod \"simulator-w4-1de75a-kserve-6887dd4b46-kc5c2\" (UID: \"926fa847-0c3c-43e0-b378-04c82433cd98\") " pod="e2e-models-e2e-worker-w4-1de75a/simulator-w4-1de75a-kserve-6887dd4b46-kc5c2" Oct 30 21:35:31.959683 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:31.959608 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/926fa847-0c3c-43e0-b378-04c82433cd98-model-cache\") pod \"simulator-w4-1de75a-kserve-6887dd4b46-kc5c2\" (UID: \"926fa847-0c3c-43e0-b378-04c82433cd98\") " pod="e2e-models-e2e-worker-w4-1de75a/simulator-w4-1de75a-kserve-6887dd4b46-kc5c2" Oct 30 21:35:31.961319 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:31.961295 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/926fa847-0c3c-43e0-b378-04c82433cd98-dshm\") pod \"simulator-w4-1de75a-kserve-6887dd4b46-kc5c2\" (UID: \"926fa847-0c3c-43e0-b378-04c82433cd98\") " pod="e2e-models-e2e-worker-w4-1de75a/simulator-w4-1de75a-kserve-6887dd4b46-kc5c2" Oct 30 21:35:31.961444 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:31.961424 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/926fa847-0c3c-43e0-b378-04c82433cd98-tls-certs\") pod \"simulator-w4-1de75a-kserve-6887dd4b46-kc5c2\" (UID: \"926fa847-0c3c-43e0-b378-04c82433cd98\") " pod="e2e-models-e2e-worker-w4-1de75a/simulator-w4-1de75a-kserve-6887dd4b46-kc5c2" Oct 30 21:35:32.021809 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:32.021713 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfqvl\" (UniqueName: \"kubernetes.io/projected/926fa847-0c3c-43e0-b378-04c82433cd98-kube-api-access-kfqvl\") pod \"simulator-w4-1de75a-kserve-6887dd4b46-kc5c2\" (UID: \"926fa847-0c3c-43e0-b378-04c82433cd98\") " pod="e2e-models-e2e-worker-w4-1de75a/simulator-w4-1de75a-kserve-6887dd4b46-kc5c2" Oct 30 21:35:32.041463 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:32.041424 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1de75a/simulator-w4-1de75a-kserve-6887dd4b46-kc5c2" Oct 30 21:35:32.420646 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:32.420624 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1de75a/simulator-w4-1de75a-kserve-6887dd4b46-kc5c2"] Oct 30 21:35:32.423218 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:35:32.423185 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod926fa847_0c3c_43e0_b378_04c82433cd98.slice/crio-b063ab1039352549a6eb78f71e89ccf72a9605c2864c0b045958912746ada33f WatchSource:0}: Error finding container b063ab1039352549a6eb78f71e89ccf72a9605c2864c0b045958912746ada33f: Status 404 returned error can't find the container with id b063ab1039352549a6eb78f71e89ccf72a9605c2864c0b045958912746ada33f Oct 30 21:35:32.753836 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:32.753748 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1de75a/simulator-w4-1de75a-kserve-6887dd4b46-kc5c2" event={"ID":"926fa847-0c3c-43e0-b378-04c82433cd98","Type":"ContainerStarted","Data":"efb462192c8e0970e0ba9df89d040031ba835fb0074cfd84ae03e3ce7349c9e6"} Oct 30 21:35:32.753836 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:32.753790 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1de75a/simulator-w4-1de75a-kserve-6887dd4b46-kc5c2" event={"ID":"926fa847-0c3c-43e0-b378-04c82433cd98","Type":"ContainerStarted","Data":"b063ab1039352549a6eb78f71e89ccf72a9605c2864c0b045958912746ada33f"} Oct 30 21:35:33.041585 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:33.041491 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1de75a/simulator-w4-1de75a-kserve-6887dd4b46-kc5c2" Oct 30 21:35:42.042066 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:42.041955 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-1de75a/simulator-w4-1de75a-kserve-6887dd4b46-kc5c2" Oct 30 21:35:42.055153 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:42.055120 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-1de75a/simulator-w4-1de75a-kserve-6887dd4b46-kc5c2" Oct 30 21:35:42.174112 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:42.174065 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-1de75a/simulator-w4-1de75a-kserve-6887dd4b46-kc5c2" podStartSLOduration=11.174051461 podStartE2EDuration="11.174051461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:32.872399386 +0000 UTC m=+716.614920320" watchObservedRunningTime="2026-10-30 21:35:42.174051461 +0000 UTC m=+725.916572396" Oct 30 21:35:42.811537 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:42.811507 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1de75a/simulator-w4-1de75a-kserve-6887dd4b46-kc5c2" Oct 30 21:35:47.243940 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:47.243887 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h"] Oct 30 21:35:47.248097 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:47.248074 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h" Oct 30 21:35:47.313928 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:47.313879 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-c7d477\"/\"openshift-service-ca.crt\"" Oct 30 21:35:47.313928 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:47.313879 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-c7d477\"/\"kube-root-ca.crt\"" Oct 30 21:35:47.352160 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:47.352122 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-c7d477\"/\"default-dockercfg-9xlfk\"" Oct 30 21:35:47.352339 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:47.352205 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-c7d477\"/\"simulator-w5-c7d477-kserve-self-signed-certs\"" Oct 30 21:35:47.353984 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:47.353959 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h"] Oct 30 21:35:47.418049 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:47.418001 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmnrq\" (UniqueName: \"kubernetes.io/projected/0a0b0425-2959-4690-9847-38852e167230-kube-api-access-cmnrq\") pod \"simulator-w5-c7d477-kserve-9f759d5bd-59r8h\" (UID: \"0a0b0425-2959-4690-9847-38852e167230\") " pod="e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h" Oct 30 21:35:47.418049 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:47.418051 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a0b0425-2959-4690-9847-38852e167230-tmp-dir\") pod \"simulator-w5-c7d477-kserve-9f759d5bd-59r8h\" (UID: \"0a0b0425-2959-4690-9847-38852e167230\") " pod="e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h" Oct 30 21:35:47.418328 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:47.418165 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a0b0425-2959-4690-9847-38852e167230-dshm\") pod \"simulator-w5-c7d477-kserve-9f759d5bd-59r8h\" (UID: \"0a0b0425-2959-4690-9847-38852e167230\") " pod="e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h" Oct 30 21:35:47.418328 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:47.418184 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a0b0425-2959-4690-9847-38852e167230-model-cache\") pod \"simulator-w5-c7d477-kserve-9f759d5bd-59r8h\" (UID: \"0a0b0425-2959-4690-9847-38852e167230\") " pod="e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h" Oct 30 21:35:47.418328 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:47.418203 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a0b0425-2959-4690-9847-38852e167230-tls-certs\") pod \"simulator-w5-c7d477-kserve-9f759d5bd-59r8h\" (UID: \"0a0b0425-2959-4690-9847-38852e167230\") " pod="e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h" Oct 30 21:35:47.418480 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:47.418344 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a0b0425-2959-4690-9847-38852e167230-home\") pod \"simulator-w5-c7d477-kserve-9f759d5bd-59r8h\" (UID: \"0a0b0425-2959-4690-9847-38852e167230\") " pod="e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h" Oct 30 21:35:47.519503 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:47.519396 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a0b0425-2959-4690-9847-38852e167230-home\") pod \"simulator-w5-c7d477-kserve-9f759d5bd-59r8h\" (UID: \"0a0b0425-2959-4690-9847-38852e167230\") " pod="e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h" Oct 30 21:35:47.519503 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:47.519477 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmnrq\" (UniqueName: \"kubernetes.io/projected/0a0b0425-2959-4690-9847-38852e167230-kube-api-access-cmnrq\") pod \"simulator-w5-c7d477-kserve-9f759d5bd-59r8h\" (UID: \"0a0b0425-2959-4690-9847-38852e167230\") " pod="e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h" Oct 30 21:35:47.519737 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:47.519504 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a0b0425-2959-4690-9847-38852e167230-tmp-dir\") pod \"simulator-w5-c7d477-kserve-9f759d5bd-59r8h\" (UID: \"0a0b0425-2959-4690-9847-38852e167230\") " pod="e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h" Oct 30 21:35:47.519737 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:47.519587 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a0b0425-2959-4690-9847-38852e167230-dshm\") pod \"simulator-w5-c7d477-kserve-9f759d5bd-59r8h\" (UID: \"0a0b0425-2959-4690-9847-38852e167230\") " pod="e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h" Oct 30 21:35:47.519737 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:47.519614 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a0b0425-2959-4690-9847-38852e167230-model-cache\") pod \"simulator-w5-c7d477-kserve-9f759d5bd-59r8h\" (UID: \"0a0b0425-2959-4690-9847-38852e167230\") " pod="e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h" Oct 30 21:35:47.519737 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:47.519644 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a0b0425-2959-4690-9847-38852e167230-tls-certs\") pod \"simulator-w5-c7d477-kserve-9f759d5bd-59r8h\" (UID: \"0a0b0425-2959-4690-9847-38852e167230\") " pod="e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h" Oct 30 21:35:47.519987 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:47.519924 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a0b0425-2959-4690-9847-38852e167230-home\") pod \"simulator-w5-c7d477-kserve-9f759d5bd-59r8h\" (UID: \"0a0b0425-2959-4690-9847-38852e167230\") " pod="e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h" Oct 30 21:35:47.520225 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:47.520200 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a0b0425-2959-4690-9847-38852e167230-tmp-dir\") pod \"simulator-w5-c7d477-kserve-9f759d5bd-59r8h\" (UID: \"0a0b0425-2959-4690-9847-38852e167230\") " pod="e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h" Oct 30 21:35:47.520547 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:47.520519 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a0b0425-2959-4690-9847-38852e167230-model-cache\") pod \"simulator-w5-c7d477-kserve-9f759d5bd-59r8h\" (UID: \"0a0b0425-2959-4690-9847-38852e167230\") " pod="e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h" Oct 30 21:35:47.523039 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:47.522980 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a0b0425-2959-4690-9847-38852e167230-dshm\") pod \"simulator-w5-c7d477-kserve-9f759d5bd-59r8h\" (UID: \"0a0b0425-2959-4690-9847-38852e167230\") " pod="e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h" Oct 30 21:35:47.523039 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:47.522986 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a0b0425-2959-4690-9847-38852e167230-tls-certs\") pod \"simulator-w5-c7d477-kserve-9f759d5bd-59r8h\" (UID: \"0a0b0425-2959-4690-9847-38852e167230\") " pod="e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h" Oct 30 21:35:47.582222 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:47.582177 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmnrq\" (UniqueName: \"kubernetes.io/projected/0a0b0425-2959-4690-9847-38852e167230-kube-api-access-cmnrq\") pod \"simulator-w5-c7d477-kserve-9f759d5bd-59r8h\" (UID: \"0a0b0425-2959-4690-9847-38852e167230\") " pod="e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h" Oct 30 21:35:47.861761 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:47.861661 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h" Oct 30 21:35:48.239033 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:48.238978 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h"] Oct 30 21:35:48.241635 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:35:48.241603 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a0b0425_2959_4690_9847_38852e167230.slice/crio-819a71e6830606f37af1fd1adcf2903b2f503912836e3f808e5234f9ce8f242c WatchSource:0}: Error finding container 819a71e6830606f37af1fd1adcf2903b2f503912836e3f808e5234f9ce8f242c: Status 404 returned error can't find the container with id 819a71e6830606f37af1fd1adcf2903b2f503912836e3f808e5234f9ce8f242c Oct 30 21:35:48.840419 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:48.840378 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h" event={"ID":"0a0b0425-2959-4690-9847-38852e167230","Type":"ContainerStarted","Data":"2cb1be98fa914075f3145977294d5c030762062ddf458e3c7cf24fa93cbbbcc9"} Oct 30 21:35:48.840419 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:48.840425 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h" event={"ID":"0a0b0425-2959-4690-9847-38852e167230","Type":"ContainerStarted","Data":"819a71e6830606f37af1fd1adcf2903b2f503912836e3f808e5234f9ce8f242c"} Oct 30 21:35:48.861931 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:48.861878 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h" Oct 30 21:35:48.968553 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:48.968497 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h" podStartSLOduration=1.968481003 podStartE2EDuration="1.968481003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:48.966681805 +0000 UTC m=+732.709202753" watchObservedRunningTime="2026-10-30 21:35:48.968481003 +0000 UTC m=+732.711002214" Oct 30 21:35:57.862085 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:57.862045 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h" Oct 30 21:35:57.874649 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:57.874621 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h" Oct 30 21:35:57.896164 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:35:57.896133 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h" Oct 30 21:36:00.144844 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:36:00.144805 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:36:04.300076 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:04.300029 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-69e4b2/simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj"] Oct 30 21:36:04.304120 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:04.304102 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-69e4b2/simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj" Oct 30 21:36:04.364667 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:04.364639 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-69e4b2\"/\"default-dockercfg-rnhwl\"" Oct 30 21:36:04.364853 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:04.364784 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-69e4b2\"/\"simulator-w6-69e4b2-kserve-self-signed-certs\"" Oct 30 21:36:04.364973 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:04.364921 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-69e4b2\"/\"openshift-service-ca.crt\"" Oct 30 21:36:04.364973 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:04.364947 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-69e4b2\"/\"kube-root-ca.crt\"" Oct 30 21:36:04.373208 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:04.373180 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-69e4b2/simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj"] Oct 30 21:36:04.382399 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:04.382367 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/698fc0f2-64ce-4e7e-b482-8ebd9f33f84d-tmp-dir\") pod \"simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj\" (UID: \"698fc0f2-64ce-4e7e-b482-8ebd9f33f84d\") " pod="e2e-models-e2e-worker-w6-69e4b2/simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj" Oct 30 21:36:04.382576 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:04.382405 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/698fc0f2-64ce-4e7e-b482-8ebd9f33f84d-tls-certs\") pod \"simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj\" (UID: \"698fc0f2-64ce-4e7e-b482-8ebd9f33f84d\") " pod="e2e-models-e2e-worker-w6-69e4b2/simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj" Oct 30 21:36:04.382576 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:04.382445 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndhwn\" (UniqueName: \"kubernetes.io/projected/698fc0f2-64ce-4e7e-b482-8ebd9f33f84d-kube-api-access-ndhwn\") pod \"simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj\" (UID: \"698fc0f2-64ce-4e7e-b482-8ebd9f33f84d\") " pod="e2e-models-e2e-worker-w6-69e4b2/simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj" Oct 30 21:36:04.382576 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:04.382565 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/698fc0f2-64ce-4e7e-b482-8ebd9f33f84d-dshm\") pod \"simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj\" (UID: \"698fc0f2-64ce-4e7e-b482-8ebd9f33f84d\") " pod="e2e-models-e2e-worker-w6-69e4b2/simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj" Oct 30 21:36:04.382696 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:04.382621 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/698fc0f2-64ce-4e7e-b482-8ebd9f33f84d-model-cache\") pod \"simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj\" (UID: \"698fc0f2-64ce-4e7e-b482-8ebd9f33f84d\") " pod="e2e-models-e2e-worker-w6-69e4b2/simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj" Oct 30 21:36:04.382737 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:04.382694 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/698fc0f2-64ce-4e7e-b482-8ebd9f33f84d-home\") pod \"simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj\" (UID: \"698fc0f2-64ce-4e7e-b482-8ebd9f33f84d\") " pod="e2e-models-e2e-worker-w6-69e4b2/simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj" Oct 30 21:36:04.483957 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:04.483893 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/698fc0f2-64ce-4e7e-b482-8ebd9f33f84d-home\") pod \"simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj\" (UID: \"698fc0f2-64ce-4e7e-b482-8ebd9f33f84d\") " pod="e2e-models-e2e-worker-w6-69e4b2/simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj" Oct 30 21:36:04.484158 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:04.483994 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/698fc0f2-64ce-4e7e-b482-8ebd9f33f84d-tmp-dir\") pod \"simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj\" (UID: \"698fc0f2-64ce-4e7e-b482-8ebd9f33f84d\") " pod="e2e-models-e2e-worker-w6-69e4b2/simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj" Oct 30 21:36:04.484158 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:04.484013 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/698fc0f2-64ce-4e7e-b482-8ebd9f33f84d-tls-certs\") pod \"simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj\" (UID: \"698fc0f2-64ce-4e7e-b482-8ebd9f33f84d\") " pod="e2e-models-e2e-worker-w6-69e4b2/simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj" Oct 30 21:36:04.484158 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:04.484045 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndhwn\" (UniqueName: \"kubernetes.io/projected/698fc0f2-64ce-4e7e-b482-8ebd9f33f84d-kube-api-access-ndhwn\") pod \"simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj\" (UID: \"698fc0f2-64ce-4e7e-b482-8ebd9f33f84d\") " pod="e2e-models-e2e-worker-w6-69e4b2/simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj" Oct 30 21:36:04.484158 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:04.484091 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/698fc0f2-64ce-4e7e-b482-8ebd9f33f84d-dshm\") pod \"simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj\" (UID: \"698fc0f2-64ce-4e7e-b482-8ebd9f33f84d\") " pod="e2e-models-e2e-worker-w6-69e4b2/simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj" Oct 30 21:36:04.484158 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:04.484113 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/698fc0f2-64ce-4e7e-b482-8ebd9f33f84d-model-cache\") pod \"simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj\" (UID: \"698fc0f2-64ce-4e7e-b482-8ebd9f33f84d\") " pod="e2e-models-e2e-worker-w6-69e4b2/simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj" Oct 30 21:36:04.484434 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:04.484301 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/698fc0f2-64ce-4e7e-b482-8ebd9f33f84d-home\") pod \"simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj\" (UID: \"698fc0f2-64ce-4e7e-b482-8ebd9f33f84d\") " pod="e2e-models-e2e-worker-w6-69e4b2/simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj" Oct 30 21:36:04.484434 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:04.484409 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/698fc0f2-64ce-4e7e-b482-8ebd9f33f84d-tmp-dir\") pod \"simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj\" (UID: \"698fc0f2-64ce-4e7e-b482-8ebd9f33f84d\") " pod="e2e-models-e2e-worker-w6-69e4b2/simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj" Oct 30 21:36:04.484558 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:04.484528 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/698fc0f2-64ce-4e7e-b482-8ebd9f33f84d-model-cache\") pod \"simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj\" (UID: \"698fc0f2-64ce-4e7e-b482-8ebd9f33f84d\") " pod="e2e-models-e2e-worker-w6-69e4b2/simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj" Oct 30 21:36:04.486368 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:04.486346 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/698fc0f2-64ce-4e7e-b482-8ebd9f33f84d-dshm\") pod \"simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj\" (UID: \"698fc0f2-64ce-4e7e-b482-8ebd9f33f84d\") " pod="e2e-models-e2e-worker-w6-69e4b2/simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj" Oct 30 21:36:04.486600 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:04.486582 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/698fc0f2-64ce-4e7e-b482-8ebd9f33f84d-tls-certs\") pod \"simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj\" (UID: \"698fc0f2-64ce-4e7e-b482-8ebd9f33f84d\") " pod="e2e-models-e2e-worker-w6-69e4b2/simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj" Oct 30 21:36:04.561789 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:04.561706 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndhwn\" (UniqueName: \"kubernetes.io/projected/698fc0f2-64ce-4e7e-b482-8ebd9f33f84d-kube-api-access-ndhwn\") pod \"simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj\" (UID: \"698fc0f2-64ce-4e7e-b482-8ebd9f33f84d\") " pod="e2e-models-e2e-worker-w6-69e4b2/simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj" Oct 30 21:36:04.618037 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:04.617996 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-69e4b2/simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj" Oct 30 21:36:04.999134 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:04.999108 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-69e4b2/simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj"] Oct 30 21:36:05.001811 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:36:05.001775 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod698fc0f2_64ce_4e7e_b482_8ebd9f33f84d.slice/crio-ca0f1ae32cd917020a6b1229fb468ef67eeb0acdc5e2c4fb524f0ce5c7c7b2fb WatchSource:0}: Error finding container ca0f1ae32cd917020a6b1229fb468ef67eeb0acdc5e2c4fb524f0ce5c7c7b2fb: Status 404 returned error can't find the container with id ca0f1ae32cd917020a6b1229fb468ef67eeb0acdc5e2c4fb524f0ce5c7c7b2fb Oct 30 21:36:05.926376 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:05.926340 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-69e4b2/simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj" event={"ID":"698fc0f2-64ce-4e7e-b482-8ebd9f33f84d","Type":"ContainerStarted","Data":"75ff557b501509dd25ee3faab1db2ba7d2b25e88e5954f1f97ae72231999acce"} Oct 30 21:36:05.926785 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:05.926384 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-69e4b2/simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj" event={"ID":"698fc0f2-64ce-4e7e-b482-8ebd9f33f84d","Type":"ContainerStarted","Data":"ca0f1ae32cd917020a6b1229fb468ef67eeb0acdc5e2c4fb524f0ce5c7c7b2fb"} Oct 30 21:36:06.046809 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:06.046755 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-69e4b2/simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj" podStartSLOduration=2.046742209 podStartE2EDuration="2.046742209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:06.045556576 +0000 UTC m=+749.788077513" watchObservedRunningTime="2026-10-30 21:36:06.046742209 +0000 UTC m=+749.789263143" Oct 30 21:36:06.618667 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:06.618629 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-69e4b2/simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj" Oct 30 21:36:14.618773 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:14.618730 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-69e4b2/simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj" Oct 30 21:36:14.631535 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:14.631507 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-69e4b2/simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj" Oct 30 21:36:14.981257 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:14.981222 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-69e4b2/simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj" Oct 30 21:36:19.824222 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:19.824176 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-691598/simulator-w0-691598-kserve-76fb5d5bc7-l5gqk"] Oct 30 21:36:19.827971 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:19.827946 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-691598/simulator-w0-691598-kserve-76fb5d5bc7-l5gqk" Oct 30 21:36:19.881603 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:19.881572 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-691598\"/\"default-dockercfg-nbw99\"" Oct 30 21:36:19.932214 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:19.932176 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-691598\"/\"openshift-service-ca.crt\"" Oct 30 21:36:19.932214 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:19.932180 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-691598\"/\"simulator-w0-691598-kserve-self-signed-certs\"" Oct 30 21:36:19.932214 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:19.932188 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-691598\"/\"kube-root-ca.crt\"" Oct 30 21:36:19.936282 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:19.936257 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-691598/simulator-w0-691598-kserve-76fb5d5bc7-l5gqk"] Oct 30 21:36:19.939482 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:19.939461 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9-model-cache\") pod \"simulator-w0-691598-kserve-76fb5d5bc7-l5gqk\" (UID: \"4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9\") " pod="e2e-models-e2e-worker-w0-691598/simulator-w0-691598-kserve-76fb5d5bc7-l5gqk" Oct 30 21:36:19.939597 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:19.939500 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9-dshm\") pod \"simulator-w0-691598-kserve-76fb5d5bc7-l5gqk\" (UID: \"4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9\") " pod="e2e-models-e2e-worker-w0-691598/simulator-w0-691598-kserve-76fb5d5bc7-l5gqk" Oct 30 21:36:19.939640 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:19.939591 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57k8q\" (UniqueName: \"kubernetes.io/projected/4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9-kube-api-access-57k8q\") pod \"simulator-w0-691598-kserve-76fb5d5bc7-l5gqk\" (UID: \"4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9\") " pod="e2e-models-e2e-worker-w0-691598/simulator-w0-691598-kserve-76fb5d5bc7-l5gqk" Oct 30 21:36:19.939681 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:19.939639 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9-tls-certs\") pod \"simulator-w0-691598-kserve-76fb5d5bc7-l5gqk\" (UID: \"4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9\") " pod="e2e-models-e2e-worker-w0-691598/simulator-w0-691598-kserve-76fb5d5bc7-l5gqk" Oct 30 21:36:19.939681 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:19.939667 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9-home\") pod \"simulator-w0-691598-kserve-76fb5d5bc7-l5gqk\" (UID: \"4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9\") " pod="e2e-models-e2e-worker-w0-691598/simulator-w0-691598-kserve-76fb5d5bc7-l5gqk" Oct 30 21:36:19.939770 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:19.939755 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9-tmp-dir\") pod \"simulator-w0-691598-kserve-76fb5d5bc7-l5gqk\" (UID: \"4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9\") " pod="e2e-models-e2e-worker-w0-691598/simulator-w0-691598-kserve-76fb5d5bc7-l5gqk" Oct 30 21:36:20.041222 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:20.041174 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9-dshm\") pod \"simulator-w0-691598-kserve-76fb5d5bc7-l5gqk\" (UID: \"4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9\") " pod="e2e-models-e2e-worker-w0-691598/simulator-w0-691598-kserve-76fb5d5bc7-l5gqk" Oct 30 21:36:20.041393 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:20.041283 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57k8q\" (UniqueName: \"kubernetes.io/projected/4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9-kube-api-access-57k8q\") pod \"simulator-w0-691598-kserve-76fb5d5bc7-l5gqk\" (UID: \"4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9\") " pod="e2e-models-e2e-worker-w0-691598/simulator-w0-691598-kserve-76fb5d5bc7-l5gqk" Oct 30 21:36:20.041393 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:20.041323 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9-tls-certs\") pod \"simulator-w0-691598-kserve-76fb5d5bc7-l5gqk\" (UID: \"4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9\") " pod="e2e-models-e2e-worker-w0-691598/simulator-w0-691598-kserve-76fb5d5bc7-l5gqk" Oct 30 21:36:20.041474 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:20.041448 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9-home\") pod \"simulator-w0-691598-kserve-76fb5d5bc7-l5gqk\" (UID: \"4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9\") " pod="e2e-models-e2e-worker-w0-691598/simulator-w0-691598-kserve-76fb5d5bc7-l5gqk" Oct 30 21:36:20.041548 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:20.041530 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9-tmp-dir\") pod \"simulator-w0-691598-kserve-76fb5d5bc7-l5gqk\" (UID: \"4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9\") " pod="e2e-models-e2e-worker-w0-691598/simulator-w0-691598-kserve-76fb5d5bc7-l5gqk" Oct 30 21:36:20.041616 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:20.041599 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9-model-cache\") pod \"simulator-w0-691598-kserve-76fb5d5bc7-l5gqk\" (UID: \"4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9\") " pod="e2e-models-e2e-worker-w0-691598/simulator-w0-691598-kserve-76fb5d5bc7-l5gqk" Oct 30 21:36:20.041819 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:20.041795 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9-home\") pod \"simulator-w0-691598-kserve-76fb5d5bc7-l5gqk\" (UID: \"4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9\") " pod="e2e-models-e2e-worker-w0-691598/simulator-w0-691598-kserve-76fb5d5bc7-l5gqk" Oct 30 21:36:20.041945 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:20.041921 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9-tmp-dir\") pod \"simulator-w0-691598-kserve-76fb5d5bc7-l5gqk\" (UID: \"4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9\") " pod="e2e-models-e2e-worker-w0-691598/simulator-w0-691598-kserve-76fb5d5bc7-l5gqk" Oct 30 21:36:20.042039 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:20.042018 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9-model-cache\") pod \"simulator-w0-691598-kserve-76fb5d5bc7-l5gqk\" (UID: \"4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9\") " pod="e2e-models-e2e-worker-w0-691598/simulator-w0-691598-kserve-76fb5d5bc7-l5gqk" Oct 30 21:36:20.043544 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:20.043518 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9-dshm\") pod \"simulator-w0-691598-kserve-76fb5d5bc7-l5gqk\" (UID: \"4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9\") " pod="e2e-models-e2e-worker-w0-691598/simulator-w0-691598-kserve-76fb5d5bc7-l5gqk" Oct 30 21:36:20.043811 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:20.043792 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9-tls-certs\") pod \"simulator-w0-691598-kserve-76fb5d5bc7-l5gqk\" (UID: \"4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9\") " pod="e2e-models-e2e-worker-w0-691598/simulator-w0-691598-kserve-76fb5d5bc7-l5gqk" Oct 30 21:36:20.103291 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:20.103204 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57k8q\" (UniqueName: \"kubernetes.io/projected/4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9-kube-api-access-57k8q\") pod \"simulator-w0-691598-kserve-76fb5d5bc7-l5gqk\" (UID: \"4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9\") " pod="e2e-models-e2e-worker-w0-691598/simulator-w0-691598-kserve-76fb5d5bc7-l5gqk" Oct 30 21:36:20.141872 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:20.141827 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-691598/simulator-w0-691598-kserve-76fb5d5bc7-l5gqk" Oct 30 21:36:20.535429 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:20.535403 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-691598/simulator-w0-691598-kserve-76fb5d5bc7-l5gqk"] Oct 30 21:36:20.538121 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:36:20.538092 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4cd6c7e1_b6ab_45fc_ac45_e9e3fded21b9.slice/crio-55821add0564dd4133dad6c0599ef8773b9c2120b3669c74e0cd5995488764c7 WatchSource:0}: Error finding container 55821add0564dd4133dad6c0599ef8773b9c2120b3669c74e0cd5995488764c7: Status 404 returned error can't find the container with id 55821add0564dd4133dad6c0599ef8773b9c2120b3669c74e0cd5995488764c7 Oct 30 21:36:21.005946 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:21.005884 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-691598/simulator-w0-691598-kserve-76fb5d5bc7-l5gqk" event={"ID":"4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9","Type":"ContainerStarted","Data":"91bb8ff3be5491a8c5744b752374ad9c8f35edae37ca8ea5f864aeecdf424d7f"} Oct 30 21:36:21.005946 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:21.005951 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-691598/simulator-w0-691598-kserve-76fb5d5bc7-l5gqk" event={"ID":"4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9","Type":"ContainerStarted","Data":"55821add0564dd4133dad6c0599ef8773b9c2120b3669c74e0cd5995488764c7"} Oct 30 21:36:21.125390 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:21.125338 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-691598/simulator-w0-691598-kserve-76fb5d5bc7-l5gqk" podStartSLOduration=2.125323108 podStartE2EDuration="2.125323108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:21.124783336 +0000 UTC m=+764.867304270" watchObservedRunningTime="2026-10-30 21:36:21.125323108 +0000 UTC m=+764.867844042" Oct 30 21:36:21.142670 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:21.142636 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-691598/simulator-w0-691598-kserve-76fb5d5bc7-l5gqk" Oct 30 21:36:30.142606 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:30.142558 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-691598/simulator-w0-691598-kserve-76fb5d5bc7-l5gqk" Oct 30 21:36:30.144848 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:36:30.144809 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:36:30.155672 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:30.155647 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-691598/simulator-w0-691598-kserve-76fb5d5bc7-l5gqk" Oct 30 21:36:31.069847 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:36:31.069820 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-691598/simulator-w0-691598-kserve-76fb5d5bc7-l5gqk" Oct 30 21:37:00.147006 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:37:00.146963 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:37:01.309688 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:01.309648 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-6ea374/simulator-w1-6ea374-kserve-775cfdf98b-p5q25"] Oct 30 21:37:01.313767 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:01.313746 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6ea374/simulator-w1-6ea374-kserve-775cfdf98b-p5q25" Oct 30 21:37:01.368874 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:01.368831 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0d53207-65f3-4d43-baba-66bc4c366885-tls-certs\") pod \"simulator-w1-6ea374-kserve-775cfdf98b-p5q25\" (UID: \"b0d53207-65f3-4d43-baba-66bc4c366885\") " pod="e2e-models-e2e-worker-w1-6ea374/simulator-w1-6ea374-kserve-775cfdf98b-p5q25" Oct 30 21:37:01.369106 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:01.368878 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0d53207-65f3-4d43-baba-66bc4c366885-dshm\") pod \"simulator-w1-6ea374-kserve-775cfdf98b-p5q25\" (UID: \"b0d53207-65f3-4d43-baba-66bc4c366885\") " pod="e2e-models-e2e-worker-w1-6ea374/simulator-w1-6ea374-kserve-775cfdf98b-p5q25" Oct 30 21:37:01.369106 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:01.369032 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0d53207-65f3-4d43-baba-66bc4c366885-tmp-dir\") pod \"simulator-w1-6ea374-kserve-775cfdf98b-p5q25\" (UID: \"b0d53207-65f3-4d43-baba-66bc4c366885\") " pod="e2e-models-e2e-worker-w1-6ea374/simulator-w1-6ea374-kserve-775cfdf98b-p5q25" Oct 30 21:37:01.369106 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:01.369070 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0d53207-65f3-4d43-baba-66bc4c366885-model-cache\") pod \"simulator-w1-6ea374-kserve-775cfdf98b-p5q25\" (UID: \"b0d53207-65f3-4d43-baba-66bc4c366885\") " pod="e2e-models-e2e-worker-w1-6ea374/simulator-w1-6ea374-kserve-775cfdf98b-p5q25" Oct 30 21:37:01.369280 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:01.369178 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0d53207-65f3-4d43-baba-66bc4c366885-home\") pod \"simulator-w1-6ea374-kserve-775cfdf98b-p5q25\" (UID: \"b0d53207-65f3-4d43-baba-66bc4c366885\") " pod="e2e-models-e2e-worker-w1-6ea374/simulator-w1-6ea374-kserve-775cfdf98b-p5q25" Oct 30 21:37:01.369280 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:01.369218 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6s28\" (UniqueName: \"kubernetes.io/projected/b0d53207-65f3-4d43-baba-66bc4c366885-kube-api-access-t6s28\") pod \"simulator-w1-6ea374-kserve-775cfdf98b-p5q25\" (UID: \"b0d53207-65f3-4d43-baba-66bc4c366885\") " pod="e2e-models-e2e-worker-w1-6ea374/simulator-w1-6ea374-kserve-775cfdf98b-p5q25" Oct 30 21:37:01.420362 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:01.420320 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-6ea374\"/\"openshift-service-ca.crt\"" Oct 30 21:37:01.420519 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:01.420395 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-6ea374\"/\"simulator-w1-6ea374-kserve-self-signed-certs\"" Oct 30 21:37:01.420519 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:01.420460 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-6ea374\"/\"default-dockercfg-pm68c\"" Oct 30 21:37:01.420637 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:01.420524 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-6ea374\"/\"kube-root-ca.crt\"" Oct 30 21:37:01.422192 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:01.422174 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-6ea374/simulator-w1-6ea374-kserve-775cfdf98b-p5q25"] Oct 30 21:37:01.470750 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:01.470697 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0d53207-65f3-4d43-baba-66bc4c366885-tls-certs\") pod \"simulator-w1-6ea374-kserve-775cfdf98b-p5q25\" (UID: \"b0d53207-65f3-4d43-baba-66bc4c366885\") " pod="e2e-models-e2e-worker-w1-6ea374/simulator-w1-6ea374-kserve-775cfdf98b-p5q25" Oct 30 21:37:01.471088 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:01.470762 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0d53207-65f3-4d43-baba-66bc4c366885-dshm\") pod \"simulator-w1-6ea374-kserve-775cfdf98b-p5q25\" (UID: \"b0d53207-65f3-4d43-baba-66bc4c366885\") " pod="e2e-models-e2e-worker-w1-6ea374/simulator-w1-6ea374-kserve-775cfdf98b-p5q25" Oct 30 21:37:01.471088 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:01.470808 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0d53207-65f3-4d43-baba-66bc4c366885-tmp-dir\") pod \"simulator-w1-6ea374-kserve-775cfdf98b-p5q25\" (UID: \"b0d53207-65f3-4d43-baba-66bc4c366885\") " pod="e2e-models-e2e-worker-w1-6ea374/simulator-w1-6ea374-kserve-775cfdf98b-p5q25" Oct 30 21:37:01.471088 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:01.470833 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0d53207-65f3-4d43-baba-66bc4c366885-model-cache\") pod \"simulator-w1-6ea374-kserve-775cfdf98b-p5q25\" (UID: \"b0d53207-65f3-4d43-baba-66bc4c366885\") " pod="e2e-models-e2e-worker-w1-6ea374/simulator-w1-6ea374-kserve-775cfdf98b-p5q25" Oct 30 21:37:01.471088 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:01.470888 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0d53207-65f3-4d43-baba-66bc4c366885-home\") pod \"simulator-w1-6ea374-kserve-775cfdf98b-p5q25\" (UID: \"b0d53207-65f3-4d43-baba-66bc4c366885\") " pod="e2e-models-e2e-worker-w1-6ea374/simulator-w1-6ea374-kserve-775cfdf98b-p5q25" Oct 30 21:37:01.471088 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:01.470938 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6s28\" (UniqueName: \"kubernetes.io/projected/b0d53207-65f3-4d43-baba-66bc4c366885-kube-api-access-t6s28\") pod \"simulator-w1-6ea374-kserve-775cfdf98b-p5q25\" (UID: \"b0d53207-65f3-4d43-baba-66bc4c366885\") " pod="e2e-models-e2e-worker-w1-6ea374/simulator-w1-6ea374-kserve-775cfdf98b-p5q25" Oct 30 21:37:01.471422 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:01.471396 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0d53207-65f3-4d43-baba-66bc4c366885-tmp-dir\") pod \"simulator-w1-6ea374-kserve-775cfdf98b-p5q25\" (UID: \"b0d53207-65f3-4d43-baba-66bc4c366885\") " pod="e2e-models-e2e-worker-w1-6ea374/simulator-w1-6ea374-kserve-775cfdf98b-p5q25" Oct 30 21:37:01.471490 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:01.471456 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0d53207-65f3-4d43-baba-66bc4c366885-model-cache\") pod \"simulator-w1-6ea374-kserve-775cfdf98b-p5q25\" (UID: \"b0d53207-65f3-4d43-baba-66bc4c366885\") " pod="e2e-models-e2e-worker-w1-6ea374/simulator-w1-6ea374-kserve-775cfdf98b-p5q25" Oct 30 21:37:01.471548 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:01.471525 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0d53207-65f3-4d43-baba-66bc4c366885-home\") pod \"simulator-w1-6ea374-kserve-775cfdf98b-p5q25\" (UID: \"b0d53207-65f3-4d43-baba-66bc4c366885\") " pod="e2e-models-e2e-worker-w1-6ea374/simulator-w1-6ea374-kserve-775cfdf98b-p5q25" Oct 30 21:37:01.473111 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:01.473090 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0d53207-65f3-4d43-baba-66bc4c366885-dshm\") pod \"simulator-w1-6ea374-kserve-775cfdf98b-p5q25\" (UID: \"b0d53207-65f3-4d43-baba-66bc4c366885\") " pod="e2e-models-e2e-worker-w1-6ea374/simulator-w1-6ea374-kserve-775cfdf98b-p5q25" Oct 30 21:37:01.473382 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:01.473364 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0d53207-65f3-4d43-baba-66bc4c366885-tls-certs\") pod \"simulator-w1-6ea374-kserve-775cfdf98b-p5q25\" (UID: \"b0d53207-65f3-4d43-baba-66bc4c366885\") " pod="e2e-models-e2e-worker-w1-6ea374/simulator-w1-6ea374-kserve-775cfdf98b-p5q25" Oct 30 21:37:01.533409 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:01.533371 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6s28\" (UniqueName: \"kubernetes.io/projected/b0d53207-65f3-4d43-baba-66bc4c366885-kube-api-access-t6s28\") pod \"simulator-w1-6ea374-kserve-775cfdf98b-p5q25\" (UID: \"b0d53207-65f3-4d43-baba-66bc4c366885\") " pod="e2e-models-e2e-worker-w1-6ea374/simulator-w1-6ea374-kserve-775cfdf98b-p5q25" Oct 30 21:37:01.626874 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:01.626775 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6ea374/simulator-w1-6ea374-kserve-775cfdf98b-p5q25" Oct 30 21:37:02.010529 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:02.010288 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-6ea374/simulator-w1-6ea374-kserve-775cfdf98b-p5q25"] Oct 30 21:37:02.012905 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:37:02.012863 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb0d53207_65f3_4d43_baba_66bc4c366885.slice/crio-a114c8a654a51aae66357f3f87ebe2e0ce50f9f3e1ccd24d56d5cd0f813af11c WatchSource:0}: Error finding container a114c8a654a51aae66357f3f87ebe2e0ce50f9f3e1ccd24d56d5cd0f813af11c: Status 404 returned error can't find the container with id a114c8a654a51aae66357f3f87ebe2e0ce50f9f3e1ccd24d56d5cd0f813af11c Oct 30 21:37:02.221743 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:02.221701 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6ea374/simulator-w1-6ea374-kserve-775cfdf98b-p5q25" event={"ID":"b0d53207-65f3-4d43-baba-66bc4c366885","Type":"ContainerStarted","Data":"d426b45d403be493a1b773daefab3470ebd05ab0be1be9a6b4577e4bdcc91a73"} Oct 30 21:37:02.221743 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:02.221750 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6ea374/simulator-w1-6ea374-kserve-775cfdf98b-p5q25" event={"ID":"b0d53207-65f3-4d43-baba-66bc4c366885","Type":"ContainerStarted","Data":"a114c8a654a51aae66357f3f87ebe2e0ce50f9f3e1ccd24d56d5cd0f813af11c"} Oct 30 21:37:02.340918 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:02.340793 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-6ea374/simulator-w1-6ea374-kserve-775cfdf98b-p5q25" podStartSLOduration=1.340774074 podStartE2EDuration="1.340774074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:02.34046508 +0000 UTC m=+806.082986016" watchObservedRunningTime="2026-10-30 21:37:02.340774074 +0000 UTC m=+806.083295006" Oct 30 21:37:02.627720 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:02.627645 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6ea374/simulator-w1-6ea374-kserve-775cfdf98b-p5q25" Oct 30 21:37:11.627492 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:11.627458 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-6ea374/simulator-w1-6ea374-kserve-775cfdf98b-p5q25" Oct 30 21:37:11.639781 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:11.639757 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-6ea374/simulator-w1-6ea374-kserve-775cfdf98b-p5q25" Oct 30 21:37:12.292002 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:12.291913 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6ea374/simulator-w1-6ea374-kserve-775cfdf98b-p5q25" Oct 30 21:37:25.547516 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:25.547473 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-69e4b2/premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b"] Oct 30 21:37:25.552836 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:25.552808 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-69e4b2/premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b" Oct 30 21:37:25.657873 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:25.657844 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-69e4b2\"/\"premium-w6-69e4b2-kserve-self-signed-certs\"" Oct 30 21:37:25.659458 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:25.659436 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-69e4b2/premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b"] Oct 30 21:37:25.719357 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:25.719324 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqnkt\" (UniqueName: \"kubernetes.io/projected/3ed43145-33bd-4c11-bb29-dc852511f6d2-kube-api-access-dqnkt\") pod \"premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b\" (UID: \"3ed43145-33bd-4c11-bb29-dc852511f6d2\") " pod="e2e-models-e2e-worker-w6-69e4b2/premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b" Oct 30 21:37:25.719357 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:25.719359 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ed43145-33bd-4c11-bb29-dc852511f6d2-tls-certs\") pod \"premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b\" (UID: \"3ed43145-33bd-4c11-bb29-dc852511f6d2\") " pod="e2e-models-e2e-worker-w6-69e4b2/premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b" Oct 30 21:37:25.719617 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:25.719393 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ed43145-33bd-4c11-bb29-dc852511f6d2-tmp-dir\") pod \"premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b\" (UID: \"3ed43145-33bd-4c11-bb29-dc852511f6d2\") " pod="e2e-models-e2e-worker-w6-69e4b2/premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b" Oct 30 21:37:25.719617 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:25.719545 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ed43145-33bd-4c11-bb29-dc852511f6d2-home\") pod \"premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b\" (UID: \"3ed43145-33bd-4c11-bb29-dc852511f6d2\") " pod="e2e-models-e2e-worker-w6-69e4b2/premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b" Oct 30 21:37:25.719719 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:25.719642 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ed43145-33bd-4c11-bb29-dc852511f6d2-dshm\") pod \"premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b\" (UID: \"3ed43145-33bd-4c11-bb29-dc852511f6d2\") " pod="e2e-models-e2e-worker-w6-69e4b2/premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b" Oct 30 21:37:25.719719 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:25.719710 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ed43145-33bd-4c11-bb29-dc852511f6d2-model-cache\") pod \"premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b\" (UID: \"3ed43145-33bd-4c11-bb29-dc852511f6d2\") " pod="e2e-models-e2e-worker-w6-69e4b2/premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b" Oct 30 21:37:25.821332 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:25.821232 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ed43145-33bd-4c11-bb29-dc852511f6d2-dshm\") pod \"premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b\" (UID: \"3ed43145-33bd-4c11-bb29-dc852511f6d2\") " pod="e2e-models-e2e-worker-w6-69e4b2/premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b" Oct 30 21:37:25.821332 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:25.821294 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ed43145-33bd-4c11-bb29-dc852511f6d2-model-cache\") pod \"premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b\" (UID: \"3ed43145-33bd-4c11-bb29-dc852511f6d2\") " pod="e2e-models-e2e-worker-w6-69e4b2/premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b" Oct 30 21:37:25.821562 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:25.821359 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqnkt\" (UniqueName: \"kubernetes.io/projected/3ed43145-33bd-4c11-bb29-dc852511f6d2-kube-api-access-dqnkt\") pod \"premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b\" (UID: \"3ed43145-33bd-4c11-bb29-dc852511f6d2\") " pod="e2e-models-e2e-worker-w6-69e4b2/premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b" Oct 30 21:37:25.821562 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:25.821388 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ed43145-33bd-4c11-bb29-dc852511f6d2-tls-certs\") pod \"premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b\" (UID: \"3ed43145-33bd-4c11-bb29-dc852511f6d2\") " pod="e2e-models-e2e-worker-w6-69e4b2/premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b" Oct 30 21:37:25.821562 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:25.821446 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ed43145-33bd-4c11-bb29-dc852511f6d2-tmp-dir\") pod \"premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b\" (UID: \"3ed43145-33bd-4c11-bb29-dc852511f6d2\") " pod="e2e-models-e2e-worker-w6-69e4b2/premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b" Oct 30 21:37:25.821562 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:25.821530 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ed43145-33bd-4c11-bb29-dc852511f6d2-home\") pod \"premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b\" (UID: \"3ed43145-33bd-4c11-bb29-dc852511f6d2\") " pod="e2e-models-e2e-worker-w6-69e4b2/premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b" Oct 30 21:37:25.821883 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:25.821843 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ed43145-33bd-4c11-bb29-dc852511f6d2-tmp-dir\") pod \"premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b\" (UID: \"3ed43145-33bd-4c11-bb29-dc852511f6d2\") " pod="e2e-models-e2e-worker-w6-69e4b2/premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b" Oct 30 21:37:25.822042 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:25.821958 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ed43145-33bd-4c11-bb29-dc852511f6d2-home\") pod \"premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b\" (UID: \"3ed43145-33bd-4c11-bb29-dc852511f6d2\") " pod="e2e-models-e2e-worker-w6-69e4b2/premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b" Oct 30 21:37:25.822042 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:25.821982 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ed43145-33bd-4c11-bb29-dc852511f6d2-model-cache\") pod \"premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b\" (UID: \"3ed43145-33bd-4c11-bb29-dc852511f6d2\") " pod="e2e-models-e2e-worker-w6-69e4b2/premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b" Oct 30 21:37:25.823646 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:25.823625 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ed43145-33bd-4c11-bb29-dc852511f6d2-dshm\") pod \"premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b\" (UID: \"3ed43145-33bd-4c11-bb29-dc852511f6d2\") " pod="e2e-models-e2e-worker-w6-69e4b2/premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b" Oct 30 21:37:25.824290 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:25.824273 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ed43145-33bd-4c11-bb29-dc852511f6d2-tls-certs\") pod \"premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b\" (UID: \"3ed43145-33bd-4c11-bb29-dc852511f6d2\") " pod="e2e-models-e2e-worker-w6-69e4b2/premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b" Oct 30 21:37:25.887185 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:25.887144 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqnkt\" (UniqueName: \"kubernetes.io/projected/3ed43145-33bd-4c11-bb29-dc852511f6d2-kube-api-access-dqnkt\") pod \"premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b\" (UID: \"3ed43145-33bd-4c11-bb29-dc852511f6d2\") " pod="e2e-models-e2e-worker-w6-69e4b2/premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b" Oct 30 21:37:26.165183 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:26.165149 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-69e4b2/premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b" Oct 30 21:37:26.543467 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:26.543441 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-69e4b2/premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b"] Oct 30 21:37:26.546202 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:37:26.546174 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ed43145_33bd_4c11_bb29_dc852511f6d2.slice/crio-dd75240afec164be7bf5ee1f8a71bfa76c29e79d92a3081a1e0cb30093e9b272 WatchSource:0}: Error finding container dd75240afec164be7bf5ee1f8a71bfa76c29e79d92a3081a1e0cb30093e9b272: Status 404 returned error can't find the container with id dd75240afec164be7bf5ee1f8a71bfa76c29e79d92a3081a1e0cb30093e9b272 Oct 30 21:37:27.357666 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:27.357625 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-69e4b2/premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b" event={"ID":"3ed43145-33bd-4c11-bb29-dc852511f6d2","Type":"ContainerStarted","Data":"4244e067e8d04b554524e70d4bfe61ef05f6d4cb815bf2c2e2c85ed425897201"} Oct 30 21:37:27.358112 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:27.357768 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-69e4b2/premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b" event={"ID":"3ed43145-33bd-4c11-bb29-dc852511f6d2","Type":"ContainerStarted","Data":"dd75240afec164be7bf5ee1f8a71bfa76c29e79d92a3081a1e0cb30093e9b272"} Oct 30 21:37:27.477269 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:27.477213 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-69e4b2/premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b" podStartSLOduration=2.477199262 podStartE2EDuration="2.477199262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:27.476434938 +0000 UTC m=+831.218955884" watchObservedRunningTime="2026-10-30 21:37:27.477199262 +0000 UTC m=+831.219720199" Oct 30 21:37:28.165906 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:28.165864 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-69e4b2/premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b" Oct 30 21:37:30.144850 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:37:30.144816 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:37:36.165855 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:36.165813 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-69e4b2/premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b" Oct 30 21:37:36.178503 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:36.178472 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-69e4b2/premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b" Oct 30 21:37:36.417327 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:36.417252 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-69e4b2/premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b" Oct 30 21:37:49.258505 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:49.258456 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf"] Oct 30 21:37:49.262175 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:49.262151 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf" Oct 30 21:37:49.315669 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:49.315633 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-d68156-openshift-default-dockercfg-wslcg\"" Oct 30 21:37:49.350462 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:49.350430 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhxml\" (UniqueName: \"kubernetes.io/projected/424d4fd4-debd-42b5-a2ab-09eefaaefa77-kube-api-access-jhxml\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf\" (UID: \"424d4fd4-debd-42b5-a2ab-09eefaaefa77\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf" Oct 30 21:37:49.350637 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:49.350475 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/424d4fd4-debd-42b5-a2ab-09eefaaefa77-istio-token\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf\" (UID: \"424d4fd4-debd-42b5-a2ab-09eefaaefa77\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf" Oct 30 21:37:49.350637 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:49.350629 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/424d4fd4-debd-42b5-a2ab-09eefaaefa77-istio-envoy\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf\" (UID: \"424d4fd4-debd-42b5-a2ab-09eefaaefa77\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf" Oct 30 21:37:49.350715 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:49.350655 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/424d4fd4-debd-42b5-a2ab-09eefaaefa77-istio-podinfo\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf\" (UID: \"424d4fd4-debd-42b5-a2ab-09eefaaefa77\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf" Oct 30 21:37:49.350793 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:49.350763 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/424d4fd4-debd-42b5-a2ab-09eefaaefa77-istiod-ca-cert\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf\" (UID: \"424d4fd4-debd-42b5-a2ab-09eefaaefa77\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf" Oct 30 21:37:49.350828 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:49.350814 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/424d4fd4-debd-42b5-a2ab-09eefaaefa77-workload-certs\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf\" (UID: \"424d4fd4-debd-42b5-a2ab-09eefaaefa77\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf" Oct 30 21:37:49.350890 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:49.350876 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/424d4fd4-debd-42b5-a2ab-09eefaaefa77-istio-data\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf\" (UID: \"424d4fd4-debd-42b5-a2ab-09eefaaefa77\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf" Oct 30 21:37:49.350992 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:49.350971 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/424d4fd4-debd-42b5-a2ab-09eefaaefa77-workload-socket\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf\" (UID: \"424d4fd4-debd-42b5-a2ab-09eefaaefa77\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf" Oct 30 21:37:49.351029 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:49.350999 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/424d4fd4-debd-42b5-a2ab-09eefaaefa77-credential-socket\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf\" (UID: \"424d4fd4-debd-42b5-a2ab-09eefaaefa77\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf" Oct 30 21:37:49.367999 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:49.367970 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf"] Oct 30 21:37:49.451744 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:49.451708 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/424d4fd4-debd-42b5-a2ab-09eefaaefa77-istio-token\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf\" (UID: \"424d4fd4-debd-42b5-a2ab-09eefaaefa77\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf" Oct 30 21:37:49.451907 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:49.451799 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/424d4fd4-debd-42b5-a2ab-09eefaaefa77-istio-envoy\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf\" (UID: \"424d4fd4-debd-42b5-a2ab-09eefaaefa77\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf" Oct 30 21:37:49.451907 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:49.451823 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/424d4fd4-debd-42b5-a2ab-09eefaaefa77-istio-podinfo\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf\" (UID: \"424d4fd4-debd-42b5-a2ab-09eefaaefa77\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf" Oct 30 21:37:49.451907 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:49.451869 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/424d4fd4-debd-42b5-a2ab-09eefaaefa77-istiod-ca-cert\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf\" (UID: \"424d4fd4-debd-42b5-a2ab-09eefaaefa77\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf" Oct 30 21:37:49.452093 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:49.451889 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/424d4fd4-debd-42b5-a2ab-09eefaaefa77-workload-certs\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf\" (UID: \"424d4fd4-debd-42b5-a2ab-09eefaaefa77\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf" Oct 30 21:37:49.452093 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:49.452051 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/424d4fd4-debd-42b5-a2ab-09eefaaefa77-istio-data\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf\" (UID: \"424d4fd4-debd-42b5-a2ab-09eefaaefa77\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf" Oct 30 21:37:49.452208 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:49.452123 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/424d4fd4-debd-42b5-a2ab-09eefaaefa77-workload-socket\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf\" (UID: \"424d4fd4-debd-42b5-a2ab-09eefaaefa77\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf" Oct 30 21:37:49.452208 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:49.452149 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/424d4fd4-debd-42b5-a2ab-09eefaaefa77-credential-socket\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf\" (UID: \"424d4fd4-debd-42b5-a2ab-09eefaaefa77\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf" Oct 30 21:37:49.452208 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:49.452188 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhxml\" (UniqueName: \"kubernetes.io/projected/424d4fd4-debd-42b5-a2ab-09eefaaefa77-kube-api-access-jhxml\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf\" (UID: \"424d4fd4-debd-42b5-a2ab-09eefaaefa77\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf" Oct 30 21:37:49.452387 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:49.452332 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/424d4fd4-debd-42b5-a2ab-09eefaaefa77-workload-certs\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf\" (UID: \"424d4fd4-debd-42b5-a2ab-09eefaaefa77\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf" Oct 30 21:37:49.452516 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:49.452486 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/424d4fd4-debd-42b5-a2ab-09eefaaefa77-istio-data\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf\" (UID: \"424d4fd4-debd-42b5-a2ab-09eefaaefa77\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf" Oct 30 21:37:49.452588 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:49.452493 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/424d4fd4-debd-42b5-a2ab-09eefaaefa77-credential-socket\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf\" (UID: \"424d4fd4-debd-42b5-a2ab-09eefaaefa77\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf" Oct 30 21:37:49.452762 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:49.452736 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/424d4fd4-debd-42b5-a2ab-09eefaaefa77-istiod-ca-cert\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf\" (UID: \"424d4fd4-debd-42b5-a2ab-09eefaaefa77\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf" Oct 30 21:37:49.452829 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:49.452743 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/424d4fd4-debd-42b5-a2ab-09eefaaefa77-workload-socket\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf\" (UID: \"424d4fd4-debd-42b5-a2ab-09eefaaefa77\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf" Oct 30 21:37:49.454109 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:49.454087 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/424d4fd4-debd-42b5-a2ab-09eefaaefa77-istio-envoy\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf\" (UID: \"424d4fd4-debd-42b5-a2ab-09eefaaefa77\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf" Oct 30 21:37:49.454433 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:49.454408 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/424d4fd4-debd-42b5-a2ab-09eefaaefa77-istio-podinfo\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf\" (UID: \"424d4fd4-debd-42b5-a2ab-09eefaaefa77\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf" Oct 30 21:37:49.514501 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:49.514426 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhxml\" (UniqueName: \"kubernetes.io/projected/424d4fd4-debd-42b5-a2ab-09eefaaefa77-kube-api-access-jhxml\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf\" (UID: \"424d4fd4-debd-42b5-a2ab-09eefaaefa77\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf" Oct 30 21:37:49.514501 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:49.514430 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/424d4fd4-debd-42b5-a2ab-09eefaaefa77-istio-token\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf\" (UID: \"424d4fd4-debd-42b5-a2ab-09eefaaefa77\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf" Oct 30 21:37:49.575596 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:49.575549 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf" Oct 30 21:37:50.005310 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:50.005257 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf"] Oct 30 21:37:50.007763 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:37:50.007735 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod424d4fd4_debd_42b5_a2ab_09eefaaefa77.slice/crio-c4cdf1b6fc40bd4eb1c6dbee68f13409676a594b4c6c0ff8c54af20c2c046b10 WatchSource:0}: Error finding container c4cdf1b6fc40bd4eb1c6dbee68f13409676a594b4c6c0ff8c54af20c2c046b10: Status 404 returned error can't find the container with id c4cdf1b6fc40bd4eb1c6dbee68f13409676a594b4c6c0ff8c54af20c2c046b10 Oct 30 21:37:50.010015 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:50.009982 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:37:50.010094 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:50.010047 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:37:50.010094 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:50.010080 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:37:50.474459 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:50.474423 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf" event={"ID":"424d4fd4-debd-42b5-a2ab-09eefaaefa77","Type":"ContainerStarted","Data":"bb68636f461e65e0fd439fcfed1a14b521d2cb95d7de9394b7bf603061e3e566"} Oct 30 21:37:50.474824 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:50.474490 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf" event={"ID":"424d4fd4-debd-42b5-a2ab-09eefaaefa77","Type":"ContainerStarted","Data":"c4cdf1b6fc40bd4eb1c6dbee68f13409676a594b4c6c0ff8c54af20c2c046b10"} Oct 30 21:37:50.576162 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:50.576126 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf" Oct 30 21:37:50.615919 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:50.604065 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf" podStartSLOduration=1.604043412 podStartE2EDuration="1.604043412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:50.601770936 +0000 UTC m=+854.344291883" watchObservedRunningTime="2026-10-30 21:37:50.604043412 +0000 UTC m=+854.346564349" Oct 30 21:37:51.577502 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:51.577465 5748 patch_prober.go:29] interesting pod/e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 21:37:51.577959 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:51.577523 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf" podUID="424d4fd4-debd-42b5-a2ab-09eefaaefa77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 21:37:52.576689 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:52.576653 5748 patch_prober.go:29] interesting pod/e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 21:37:52.576879 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:52.576709 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf" podUID="424d4fd4-debd-42b5-a2ab-09eefaaefa77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 21:37:53.576469 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:53.576431 5748 patch_prober.go:29] interesting pod/e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 21:37:53.576865 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:53.576487 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf" podUID="424d4fd4-debd-42b5-a2ab-09eefaaefa77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 21:37:54.576607 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:54.576570 5748 patch_prober.go:29] interesting pod/e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 21:37:54.577033 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:54.576628 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf" podUID="424d4fd4-debd-42b5-a2ab-09eefaaefa77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 21:37:55.576410 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:55.576375 5748 patch_prober.go:29] interesting pod/e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 21:37:55.576618 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:55.576430 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf" podUID="424d4fd4-debd-42b5-a2ab-09eefaaefa77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 21:37:56.730394 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:56.730345 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf" Oct 30 21:37:56.731228 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:56.731203 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf" Oct 30 21:37:56.732016 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:37:56.731994 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf" Oct 30 21:38:00.147971 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:38:00.147936 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:38:23.554015 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:23.553978 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0857c785-bt6s8"] Oct 30 21:38:23.557832 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:23.557814 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0857c785-bt6s8" Oct 30 21:38:23.614100 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:23.613813 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:38:23.615281 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:23.615247 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fee336b-4d8c-4696-a812-01e6dbbd629d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0857c785-bt6s8\" (UID: \"8fee336b-4d8c-4696-a812-01e6dbbd629d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0857c785-bt6s8" Oct 30 21:38:23.615471 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:23.615454 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvkvj\" (UniqueName: \"kubernetes.io/projected/8fee336b-4d8c-4696-a812-01e6dbbd629d-kube-api-access-rvkvj\") pod \"maas-api-revoke-sub-0857c785-bt6s8\" (UID: \"8fee336b-4d8c-4696-a812-01e6dbbd629d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0857c785-bt6s8" Oct 30 21:38:23.615537 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:23.615510 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fee336b-4d8c-4696-a812-01e6dbbd629d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0857c785-bt6s8\" (UID: \"8fee336b-4d8c-4696-a812-01e6dbbd629d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0857c785-bt6s8" Oct 30 21:38:23.664369 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:23.664333 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:38:23.664369 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:23.664354 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:38:23.664635 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:23.664385 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 21:38:23.666276 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:23.666252 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0857c785-bt6s8"] Oct 30 21:38:23.716425 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:23.716387 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvkvj\" (UniqueName: \"kubernetes.io/projected/8fee336b-4d8c-4696-a812-01e6dbbd629d-kube-api-access-rvkvj\") pod \"maas-api-revoke-sub-0857c785-bt6s8\" (UID: \"8fee336b-4d8c-4696-a812-01e6dbbd629d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0857c785-bt6s8" Oct 30 21:38:23.716602 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:23.716451 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fee336b-4d8c-4696-a812-01e6dbbd629d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0857c785-bt6s8\" (UID: \"8fee336b-4d8c-4696-a812-01e6dbbd629d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0857c785-bt6s8" Oct 30 21:38:23.716602 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:23.716526 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fee336b-4d8c-4696-a812-01e6dbbd629d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0857c785-bt6s8\" (UID: \"8fee336b-4d8c-4696-a812-01e6dbbd629d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0857c785-bt6s8" Oct 30 21:38:23.717232 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:23.717208 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fee336b-4d8c-4696-a812-01e6dbbd629d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0857c785-bt6s8\" (UID: \"8fee336b-4d8c-4696-a812-01e6dbbd629d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0857c785-bt6s8" Oct 30 21:38:23.718874 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:23.718851 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fee336b-4d8c-4696-a812-01e6dbbd629d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0857c785-bt6s8\" (UID: \"8fee336b-4d8c-4696-a812-01e6dbbd629d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0857c785-bt6s8" Oct 30 21:38:23.778403 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:23.778369 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvkvj\" (UniqueName: \"kubernetes.io/projected/8fee336b-4d8c-4696-a812-01e6dbbd629d-kube-api-access-rvkvj\") pod \"maas-api-revoke-sub-0857c785-bt6s8\" (UID: \"8fee336b-4d8c-4696-a812-01e6dbbd629d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0857c785-bt6s8" Oct 30 21:38:23.871397 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:23.871320 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0857c785-bt6s8" Oct 30 21:38:24.248500 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:24.248472 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0857c785-bt6s8"] Oct 30 21:38:24.250805 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:38:24.250773 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fee336b_4d8c_4696_a812_01e6dbbd629d.slice/crio-a1b9d1fa9e33314c70d8168ef0e3bbeb358ac5345cae7f8bce5fe5cc868a21ba WatchSource:0}: Error finding container a1b9d1fa9e33314c70d8168ef0e3bbeb358ac5345cae7f8bce5fe5cc868a21ba: Status 404 returned error can't find the container with id a1b9d1fa9e33314c70d8168ef0e3bbeb358ac5345cae7f8bce5fe5cc868a21ba Oct 30 21:38:24.650923 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:24.650862 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0857c785-bt6s8" event={"ID":"8fee336b-4d8c-4696-a812-01e6dbbd629d","Type":"ContainerStarted","Data":"a1b9d1fa9e33314c70d8168ef0e3bbeb358ac5345cae7f8bce5fe5cc868a21ba"} Oct 30 21:38:28.673630 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:28.673595 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0857c785-bt6s8" event={"ID":"8fee336b-4d8c-4696-a812-01e6dbbd629d","Type":"ContainerStarted","Data":"51a5f8993a72b75d95dd38093caadd19c59a53a10e4d1da3d3b518b54f6da0d5"} Oct 30 21:38:29.679754 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:29.679721 5748 generic.go:356] "Generic (PLEG): container finished" podID="8fee336b-4d8c-4696-a812-01e6dbbd629d" containerID="51a5f8993a72b75d95dd38093caadd19c59a53a10e4d1da3d3b518b54f6da0d5" exitCode=0 Oct 30 21:38:29.680228 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:29.679816 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0857c785-bt6s8" event={"ID":"8fee336b-4d8c-4696-a812-01e6dbbd629d","Type":"ContainerDied","Data":"51a5f8993a72b75d95dd38093caadd19c59a53a10e4d1da3d3b518b54f6da0d5"} Oct 30 21:38:30.144763 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:38:30.144727 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:38:30.821309 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:30.821282 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0857c785-bt6s8" Oct 30 21:38:30.886955 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:30.886920 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fee336b-4d8c-4696-a812-01e6dbbd629d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fee336b-4d8c-4696-a812-01e6dbbd629d-maas-api-openshift-ca\") pod \"8fee336b-4d8c-4696-a812-01e6dbbd629d\" (UID: \"8fee336b-4d8c-4696-a812-01e6dbbd629d\") " Oct 30 21:38:30.887137 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:30.887007 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fee336b-4d8c-4696-a812-01e6dbbd629d-kube-api-access-rvkvj\" (UniqueName: \"kubernetes.io/projected/8fee336b-4d8c-4696-a812-01e6dbbd629d-kube-api-access-rvkvj\") pod \"8fee336b-4d8c-4696-a812-01e6dbbd629d\" (UID: \"8fee336b-4d8c-4696-a812-01e6dbbd629d\") " Oct 30 21:38:30.887137 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:30.887089 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fee336b-4d8c-4696-a812-01e6dbbd629d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fee336b-4d8c-4696-a812-01e6dbbd629d-maas-api-xks-ca\") pod \"8fee336b-4d8c-4696-a812-01e6dbbd629d\" (UID: \"8fee336b-4d8c-4696-a812-01e6dbbd629d\") " Oct 30 21:38:30.887307 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:30.887280 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fee336b-4d8c-4696-a812-01e6dbbd629d-maas-api-openshift-ca" pod "8fee336b-4d8c-4696-a812-01e6dbbd629d" (UID: "8fee336b-4d8c-4696-a812-01e6dbbd629d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:38:30.887438 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:30.887421 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fee336b-4d8c-4696-a812-01e6dbbd629d-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:30.889287 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:30.889254 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fee336b-4d8c-4696-a812-01e6dbbd629d-kube-api-access-rvkvj" pod "8fee336b-4d8c-4696-a812-01e6dbbd629d" (UID: "8fee336b-4d8c-4696-a812-01e6dbbd629d"). InnerVolumeSpecName "kube-api-access-rvkvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:38:30.889403 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:30.889317 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fee336b-4d8c-4696-a812-01e6dbbd629d-maas-api-xks-ca" pod "8fee336b-4d8c-4696-a812-01e6dbbd629d" (UID: "8fee336b-4d8c-4696-a812-01e6dbbd629d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:38:30.988424 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:30.988351 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fee336b-4d8c-4696-a812-01e6dbbd629d-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:30.988424 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:30.988381 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvkvj\" (UniqueName: \"kubernetes.io/projected/8fee336b-4d8c-4696-a812-01e6dbbd629d-kube-api-access-rvkvj\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:31.691562 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:31.691522 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0857c785-bt6s8" event={"ID":"8fee336b-4d8c-4696-a812-01e6dbbd629d","Type":"ContainerDied","Data":"a1b9d1fa9e33314c70d8168ef0e3bbeb358ac5345cae7f8bce5fe5cc868a21ba"} Oct 30 21:38:31.691744 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:31.691580 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1b9d1fa9e33314c70d8168ef0e3bbeb358ac5345cae7f8bce5fe5cc868a21ba" Oct 30 21:38:31.691744 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:31.691546 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0857c785-bt6s8" Oct 30 21:38:36.510216 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:36.510185 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j4v8r_3df036ff-1c00-4ce2-9e86-8acbeedf6c82/ovn-acl-logging/0.log" Oct 30 21:38:36.510741 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:36.510720 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j4v8r_3df036ff-1c00-4ce2-9e86-8acbeedf6c82/ovn-acl-logging/0.log" Oct 30 21:38:42.989150 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:42.989061 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9072d090-xgmg2"] Oct 30 21:38:42.989739 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:42.989717 5748 cpu_manager.go:414] "containerMap: removing container" podUID="8fee336b-4d8c-4696-a812-01e6dbbd629d" containerName="revoke-keys" Oct 30 21:38:42.989836 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:42.989742 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fee336b-4d8c-4696-a812-01e6dbbd629d" containerName="revoke-keys" Oct 30 21:38:42.989892 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:42.989864 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fee336b-4d8c-4696-a812-01e6dbbd629d" containerName="revoke-keys" Oct 30 21:38:42.993335 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:42.993314 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9072d090-xgmg2" Oct 30 21:38:43.046275 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:43.046245 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:38:43.096865 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:43.096833 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:38:43.097063 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:43.096927 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 21:38:43.098738 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:43.098718 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9072d090-xgmg2"] Oct 30 21:38:43.100505 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:43.100483 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc6702d8-x8np6"] Oct 30 21:38:43.104304 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:43.104289 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc6702d8-x8np6" Oct 30 21:38:43.109961 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:43.109937 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c5tt\" (UniqueName: \"kubernetes.io/projected/cba56321-fa3c-4c0e-a7d7-7f89d9dcf866-kube-api-access-6c5tt\") pod \"maas-api-revoke-sub-9072d090-xgmg2\" (UID: \"cba56321-fa3c-4c0e-a7d7-7f89d9dcf866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9072d090-xgmg2" Oct 30 21:38:43.110040 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:43.109997 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cba56321-fa3c-4c0e-a7d7-7f89d9dcf866-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9072d090-xgmg2\" (UID: \"cba56321-fa3c-4c0e-a7d7-7f89d9dcf866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9072d090-xgmg2" Oct 30 21:38:43.110177 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:43.110156 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cba56321-fa3c-4c0e-a7d7-7f89d9dcf866-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9072d090-xgmg2\" (UID: \"cba56321-fa3c-4c0e-a7d7-7f89d9dcf866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9072d090-xgmg2" Oct 30 21:38:43.147432 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:43.147391 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:38:43.164239 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:43.164209 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc6702d8-x8np6"] Oct 30 21:38:43.210888 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:43.210856 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c5tt\" (UniqueName: \"kubernetes.io/projected/cba56321-fa3c-4c0e-a7d7-7f89d9dcf866-kube-api-access-6c5tt\") pod \"maas-api-revoke-sub-9072d090-xgmg2\" (UID: \"cba56321-fa3c-4c0e-a7d7-7f89d9dcf866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9072d090-xgmg2" Oct 30 21:38:43.211085 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:43.210920 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38111ab1-6d6e-4392-8fd6-8fa44bbdf851-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc6702d8-x8np6\" (UID: \"38111ab1-6d6e-4392-8fd6-8fa44bbdf851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc6702d8-x8np6" Oct 30 21:38:43.211085 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:43.210943 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w94lw\" (UniqueName: \"kubernetes.io/projected/38111ab1-6d6e-4392-8fd6-8fa44bbdf851-kube-api-access-w94lw\") pod \"maas-api-revoke-sub-cc6702d8-x8np6\" (UID: \"38111ab1-6d6e-4392-8fd6-8fa44bbdf851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc6702d8-x8np6" Oct 30 21:38:43.211085 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:43.210964 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cba56321-fa3c-4c0e-a7d7-7f89d9dcf866-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9072d090-xgmg2\" (UID: \"cba56321-fa3c-4c0e-a7d7-7f89d9dcf866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9072d090-xgmg2" Oct 30 21:38:43.211085 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:43.211008 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38111ab1-6d6e-4392-8fd6-8fa44bbdf851-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc6702d8-x8np6\" (UID: \"38111ab1-6d6e-4392-8fd6-8fa44bbdf851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc6702d8-x8np6" Oct 30 21:38:43.211255 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:43.211230 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cba56321-fa3c-4c0e-a7d7-7f89d9dcf866-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9072d090-xgmg2\" (UID: \"cba56321-fa3c-4c0e-a7d7-7f89d9dcf866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9072d090-xgmg2" Oct 30 21:38:43.211801 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:43.211776 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cba56321-fa3c-4c0e-a7d7-7f89d9dcf866-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9072d090-xgmg2\" (UID: \"cba56321-fa3c-4c0e-a7d7-7f89d9dcf866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9072d090-xgmg2" Oct 30 21:38:43.213348 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:43.213331 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cba56321-fa3c-4c0e-a7d7-7f89d9dcf866-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9072d090-xgmg2\" (UID: \"cba56321-fa3c-4c0e-a7d7-7f89d9dcf866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9072d090-xgmg2" Oct 30 21:38:43.299309 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:43.299245 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c5tt\" (UniqueName: \"kubernetes.io/projected/cba56321-fa3c-4c0e-a7d7-7f89d9dcf866-kube-api-access-6c5tt\") pod \"maas-api-revoke-sub-9072d090-xgmg2\" (UID: \"cba56321-fa3c-4c0e-a7d7-7f89d9dcf866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9072d090-xgmg2" Oct 30 21:38:43.306096 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:43.306078 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9072d090-xgmg2" Oct 30 21:38:43.312504 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:43.312485 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38111ab1-6d6e-4392-8fd6-8fa44bbdf851-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc6702d8-x8np6\" (UID: \"38111ab1-6d6e-4392-8fd6-8fa44bbdf851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc6702d8-x8np6" Oct 30 21:38:43.312574 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:43.312515 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w94lw\" (UniqueName: \"kubernetes.io/projected/38111ab1-6d6e-4392-8fd6-8fa44bbdf851-kube-api-access-w94lw\") pod \"maas-api-revoke-sub-cc6702d8-x8np6\" (UID: \"38111ab1-6d6e-4392-8fd6-8fa44bbdf851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc6702d8-x8np6" Oct 30 21:38:43.312574 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:43.312569 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38111ab1-6d6e-4392-8fd6-8fa44bbdf851-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc6702d8-x8np6\" (UID: \"38111ab1-6d6e-4392-8fd6-8fa44bbdf851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc6702d8-x8np6" Oct 30 21:38:43.313131 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:43.313109 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38111ab1-6d6e-4392-8fd6-8fa44bbdf851-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc6702d8-x8np6\" (UID: \"38111ab1-6d6e-4392-8fd6-8fa44bbdf851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc6702d8-x8np6" Oct 30 21:38:43.314786 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:43.314761 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38111ab1-6d6e-4392-8fd6-8fa44bbdf851-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc6702d8-x8np6\" (UID: \"38111ab1-6d6e-4392-8fd6-8fa44bbdf851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc6702d8-x8np6" Oct 30 21:38:43.377472 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:43.377440 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w94lw\" (UniqueName: \"kubernetes.io/projected/38111ab1-6d6e-4392-8fd6-8fa44bbdf851-kube-api-access-w94lw\") pod \"maas-api-revoke-sub-cc6702d8-x8np6\" (UID: \"38111ab1-6d6e-4392-8fd6-8fa44bbdf851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc6702d8-x8np6" Oct 30 21:38:43.417566 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:43.417536 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc6702d8-x8np6" Oct 30 21:38:43.678657 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:43.678631 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9072d090-xgmg2"] Oct 30 21:38:43.680968 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:38:43.680940 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcba56321_fa3c_4c0e_a7d7_7f89d9dcf866.slice/crio-09bad658c6fb083c89ca867ed6b6668cf8884b7ecc79170eb3b80448fd6390b7 WatchSource:0}: Error finding container 09bad658c6fb083c89ca867ed6b6668cf8884b7ecc79170eb3b80448fd6390b7: Status 404 returned error can't find the container with id 09bad658c6fb083c89ca867ed6b6668cf8884b7ecc79170eb3b80448fd6390b7 Oct 30 21:38:43.755487 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:43.755452 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9072d090-xgmg2" event={"ID":"cba56321-fa3c-4c0e-a7d7-7f89d9dcf866","Type":"ContainerStarted","Data":"09bad658c6fb083c89ca867ed6b6668cf8884b7ecc79170eb3b80448fd6390b7"} Oct 30 21:38:43.794925 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:43.794883 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc6702d8-x8np6"] Oct 30 21:38:43.797730 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:38:43.797697 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38111ab1_6d6e_4392_8fd6_8fa44bbdf851.slice/crio-0d7ac4857554002312601ca7f6c0d89c21ab1b242c4a150dd9531ac46519a6de WatchSource:0}: Error finding container 0d7ac4857554002312601ca7f6c0d89c21ab1b242c4a150dd9531ac46519a6de: Status 404 returned error can't find the container with id 0d7ac4857554002312601ca7f6c0d89c21ab1b242c4a150dd9531ac46519a6de Oct 30 21:38:44.627922 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:44.627879 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef54ebcb-29n9k"] Oct 30 21:38:44.631736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:44.631716 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef54ebcb-29n9k" Oct 30 21:38:44.690119 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:44.690087 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef54ebcb-29n9k"] Oct 30 21:38:44.727533 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:44.727502 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70a7ae14-0130-4733-ae4b-e40ce0597ac9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef54ebcb-29n9k\" (UID: \"70a7ae14-0130-4733-ae4b-e40ce0597ac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef54ebcb-29n9k" Oct 30 21:38:44.727687 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:44.727656 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70a7ae14-0130-4733-ae4b-e40ce0597ac9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef54ebcb-29n9k\" (UID: \"70a7ae14-0130-4733-ae4b-e40ce0597ac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef54ebcb-29n9k" Oct 30 21:38:44.727735 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:44.727698 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4chp\" (UniqueName: \"kubernetes.io/projected/70a7ae14-0130-4733-ae4b-e40ce0597ac9-kube-api-access-n4chp\") pod \"maas-api-revoke-sub-ef54ebcb-29n9k\" (UID: \"70a7ae14-0130-4733-ae4b-e40ce0597ac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef54ebcb-29n9k" Oct 30 21:38:44.762983 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:44.762952 5748 generic.go:356] "Generic (PLEG): container finished" podID="38111ab1-6d6e-4392-8fd6-8fa44bbdf851" containerID="77d592c5ea301a2e40fc8ff01bac5fe51567ffccd094ccdfc139f6d37e1bd395" exitCode=0 Oct 30 21:38:44.763122 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:44.763039 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc6702d8-x8np6" event={"ID":"38111ab1-6d6e-4392-8fd6-8fa44bbdf851","Type":"ContainerDied","Data":"77d592c5ea301a2e40fc8ff01bac5fe51567ffccd094ccdfc139f6d37e1bd395"} Oct 30 21:38:44.763122 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:44.763089 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc6702d8-x8np6" event={"ID":"38111ab1-6d6e-4392-8fd6-8fa44bbdf851","Type":"ContainerStarted","Data":"0d7ac4857554002312601ca7f6c0d89c21ab1b242c4a150dd9531ac46519a6de"} Oct 30 21:38:44.764706 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:44.764669 5748 generic.go:356] "Generic (PLEG): container finished" podID="cba56321-fa3c-4c0e-a7d7-7f89d9dcf866" containerID="1716de9d85339349e8e86dd11cdddf865ac511264472f19054fd5ab75af00296" exitCode=0 Oct 30 21:38:44.764832 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:44.764724 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9072d090-xgmg2" event={"ID":"cba56321-fa3c-4c0e-a7d7-7f89d9dcf866","Type":"ContainerDied","Data":"1716de9d85339349e8e86dd11cdddf865ac511264472f19054fd5ab75af00296"} Oct 30 21:38:44.829160 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:44.829088 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70a7ae14-0130-4733-ae4b-e40ce0597ac9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef54ebcb-29n9k\" (UID: \"70a7ae14-0130-4733-ae4b-e40ce0597ac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef54ebcb-29n9k" Oct 30 21:38:44.829160 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:44.829126 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4chp\" (UniqueName: \"kubernetes.io/projected/70a7ae14-0130-4733-ae4b-e40ce0597ac9-kube-api-access-n4chp\") pod \"maas-api-revoke-sub-ef54ebcb-29n9k\" (UID: \"70a7ae14-0130-4733-ae4b-e40ce0597ac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef54ebcb-29n9k" Oct 30 21:38:44.829364 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:44.829179 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70a7ae14-0130-4733-ae4b-e40ce0597ac9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef54ebcb-29n9k\" (UID: \"70a7ae14-0130-4733-ae4b-e40ce0597ac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef54ebcb-29n9k" Oct 30 21:38:44.829764 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:44.829739 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70a7ae14-0130-4733-ae4b-e40ce0597ac9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef54ebcb-29n9k\" (UID: \"70a7ae14-0130-4733-ae4b-e40ce0597ac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef54ebcb-29n9k" Oct 30 21:38:44.831522 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:44.831498 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70a7ae14-0130-4733-ae4b-e40ce0597ac9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef54ebcb-29n9k\" (UID: \"70a7ae14-0130-4733-ae4b-e40ce0597ac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef54ebcb-29n9k" Oct 30 21:38:44.891170 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:44.891138 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4chp\" (UniqueName: \"kubernetes.io/projected/70a7ae14-0130-4733-ae4b-e40ce0597ac9-kube-api-access-n4chp\") pod \"maas-api-revoke-sub-ef54ebcb-29n9k\" (UID: \"70a7ae14-0130-4733-ae4b-e40ce0597ac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef54ebcb-29n9k" Oct 30 21:38:44.959532 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:44.959500 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef54ebcb-29n9k" Oct 30 21:38:45.333676 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:45.333650 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef54ebcb-29n9k"] Oct 30 21:38:45.336008 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:38:45.335982 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70a7ae14_0130_4733_ae4b_e40ce0597ac9.slice/crio-cafddd124ea6334b40310a390670fb109ef2add131d1e558cea1252a29fd4196 WatchSource:0}: Error finding container cafddd124ea6334b40310a390670fb109ef2add131d1e558cea1252a29fd4196: Status 404 returned error can't find the container with id cafddd124ea6334b40310a390670fb109ef2add131d1e558cea1252a29fd4196 Oct 30 21:38:45.772366 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:45.772325 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef54ebcb-29n9k" event={"ID":"70a7ae14-0130-4733-ae4b-e40ce0597ac9","Type":"ContainerStarted","Data":"41f99d8eac3e86214cc7c9ee2890487405b9742401266e6d314342726ad7a849"} Oct 30 21:38:45.772780 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:45.772388 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef54ebcb-29n9k" event={"ID":"70a7ae14-0130-4733-ae4b-e40ce0597ac9","Type":"ContainerStarted","Data":"cafddd124ea6334b40310a390670fb109ef2add131d1e558cea1252a29fd4196"} Oct 30 21:38:45.891714 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:45.891654 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef54ebcb-29n9k" podStartSLOduration=1.891634797 podStartE2EDuration="1.891634797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:45.889566693 +0000 UTC m=+909.632087629" watchObservedRunningTime="2026-10-30 21:38:45.891634797 +0000 UTC m=+909.634155739" Oct 30 21:38:45.924377 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:45.924349 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc6702d8-x8np6" Oct 30 21:38:45.946379 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:45.946355 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9072d090-xgmg2" Oct 30 21:38:46.040577 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:46.040548 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cba56321-fa3c-4c0e-a7d7-7f89d9dcf866-kube-api-access-6c5tt\" (UniqueName: \"kubernetes.io/projected/cba56321-fa3c-4c0e-a7d7-7f89d9dcf866-kube-api-access-6c5tt\") pod \"cba56321-fa3c-4c0e-a7d7-7f89d9dcf866\" (UID: \"cba56321-fa3c-4c0e-a7d7-7f89d9dcf866\") " Oct 30 21:38:46.040730 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:46.040605 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38111ab1-6d6e-4392-8fd6-8fa44bbdf851-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38111ab1-6d6e-4392-8fd6-8fa44bbdf851-maas-api-xks-ca\") pod \"38111ab1-6d6e-4392-8fd6-8fa44bbdf851\" (UID: \"38111ab1-6d6e-4392-8fd6-8fa44bbdf851\") " Oct 30 21:38:46.040730 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:46.040642 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cba56321-fa3c-4c0e-a7d7-7f89d9dcf866-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cba56321-fa3c-4c0e-a7d7-7f89d9dcf866-maas-api-xks-ca\") pod \"cba56321-fa3c-4c0e-a7d7-7f89d9dcf866\" (UID: \"cba56321-fa3c-4c0e-a7d7-7f89d9dcf866\") " Oct 30 21:38:46.040730 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:46.040674 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cba56321-fa3c-4c0e-a7d7-7f89d9dcf866-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cba56321-fa3c-4c0e-a7d7-7f89d9dcf866-maas-api-openshift-ca\") pod \"cba56321-fa3c-4c0e-a7d7-7f89d9dcf866\" (UID: \"cba56321-fa3c-4c0e-a7d7-7f89d9dcf866\") " Oct 30 21:38:46.040730 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:46.040700 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38111ab1-6d6e-4392-8fd6-8fa44bbdf851-kube-api-access-w94lw\" (UniqueName: \"kubernetes.io/projected/38111ab1-6d6e-4392-8fd6-8fa44bbdf851-kube-api-access-w94lw\") pod \"38111ab1-6d6e-4392-8fd6-8fa44bbdf851\" (UID: \"38111ab1-6d6e-4392-8fd6-8fa44bbdf851\") " Oct 30 21:38:46.040997 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:46.040783 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38111ab1-6d6e-4392-8fd6-8fa44bbdf851-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38111ab1-6d6e-4392-8fd6-8fa44bbdf851-maas-api-openshift-ca\") pod \"38111ab1-6d6e-4392-8fd6-8fa44bbdf851\" (UID: \"38111ab1-6d6e-4392-8fd6-8fa44bbdf851\") " Oct 30 21:38:46.041168 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:46.041090 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cba56321-fa3c-4c0e-a7d7-7f89d9dcf866-maas-api-openshift-ca" pod "cba56321-fa3c-4c0e-a7d7-7f89d9dcf866" (UID: "cba56321-fa3c-4c0e-a7d7-7f89d9dcf866"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:38:46.041285 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:46.041256 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38111ab1-6d6e-4392-8fd6-8fa44bbdf851-maas-api-openshift-ca" pod "38111ab1-6d6e-4392-8fd6-8fa44bbdf851" (UID: "38111ab1-6d6e-4392-8fd6-8fa44bbdf851"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:38:46.044002 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:46.043975 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38111ab1-6d6e-4392-8fd6-8fa44bbdf851-kube-api-access-w94lw" pod "38111ab1-6d6e-4392-8fd6-8fa44bbdf851" (UID: "38111ab1-6d6e-4392-8fd6-8fa44bbdf851"). InnerVolumeSpecName "kube-api-access-w94lw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:38:46.044103 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:46.043995 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cba56321-fa3c-4c0e-a7d7-7f89d9dcf866-maas-api-xks-ca" pod "cba56321-fa3c-4c0e-a7d7-7f89d9dcf866" (UID: "cba56321-fa3c-4c0e-a7d7-7f89d9dcf866"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:38:46.044103 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:46.044002 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cba56321-fa3c-4c0e-a7d7-7f89d9dcf866-kube-api-access-6c5tt" pod "cba56321-fa3c-4c0e-a7d7-7f89d9dcf866" (UID: "cba56321-fa3c-4c0e-a7d7-7f89d9dcf866"). InnerVolumeSpecName "kube-api-access-6c5tt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:38:46.044103 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:46.044028 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38111ab1-6d6e-4392-8fd6-8fa44bbdf851-maas-api-xks-ca" pod "38111ab1-6d6e-4392-8fd6-8fa44bbdf851" (UID: "38111ab1-6d6e-4392-8fd6-8fa44bbdf851"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:38:46.142273 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:46.142191 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c5tt\" (UniqueName: \"kubernetes.io/projected/cba56321-fa3c-4c0e-a7d7-7f89d9dcf866-kube-api-access-6c5tt\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:46.142273 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:46.142221 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38111ab1-6d6e-4392-8fd6-8fa44bbdf851-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:46.142273 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:46.142231 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cba56321-fa3c-4c0e-a7d7-7f89d9dcf866-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:46.142273 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:46.142242 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cba56321-fa3c-4c0e-a7d7-7f89d9dcf866-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:46.142273 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:46.142252 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w94lw\" (UniqueName: \"kubernetes.io/projected/38111ab1-6d6e-4392-8fd6-8fa44bbdf851-kube-api-access-w94lw\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:46.142273 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:46.142260 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38111ab1-6d6e-4392-8fd6-8fa44bbdf851-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:46.779387 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:46.779295 5748 generic.go:356] "Generic (PLEG): container finished" podID="70a7ae14-0130-4733-ae4b-e40ce0597ac9" containerID="41f99d8eac3e86214cc7c9ee2890487405b9742401266e6d314342726ad7a849" exitCode=0 Oct 30 21:38:46.779387 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:46.779331 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef54ebcb-29n9k" event={"ID":"70a7ae14-0130-4733-ae4b-e40ce0597ac9","Type":"ContainerDied","Data":"41f99d8eac3e86214cc7c9ee2890487405b9742401266e6d314342726ad7a849"} Oct 30 21:38:46.780709 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:46.780687 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9072d090-xgmg2" Oct 30 21:38:46.780818 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:46.780693 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9072d090-xgmg2" event={"ID":"cba56321-fa3c-4c0e-a7d7-7f89d9dcf866","Type":"ContainerDied","Data":"09bad658c6fb083c89ca867ed6b6668cf8884b7ecc79170eb3b80448fd6390b7"} Oct 30 21:38:46.780818 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:46.780813 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09bad658c6fb083c89ca867ed6b6668cf8884b7ecc79170eb3b80448fd6390b7" Oct 30 21:38:46.782200 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:46.782166 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc6702d8-x8np6" event={"ID":"38111ab1-6d6e-4392-8fd6-8fa44bbdf851","Type":"ContainerDied","Data":"0d7ac4857554002312601ca7f6c0d89c21ab1b242c4a150dd9531ac46519a6de"} Oct 30 21:38:46.782309 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:46.782225 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d7ac4857554002312601ca7f6c0d89c21ab1b242c4a150dd9531ac46519a6de" Oct 30 21:38:46.782309 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:46.782178 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc6702d8-x8np6" Oct 30 21:38:47.926699 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:47.926675 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef54ebcb-29n9k" Oct 30 21:38:48.061408 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:48.061326 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70a7ae14-0130-4733-ae4b-e40ce0597ac9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70a7ae14-0130-4733-ae4b-e40ce0597ac9-maas-api-openshift-ca\") pod \"70a7ae14-0130-4733-ae4b-e40ce0597ac9\" (UID: \"70a7ae14-0130-4733-ae4b-e40ce0597ac9\") " Oct 30 21:38:48.061582 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:48.061466 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70a7ae14-0130-4733-ae4b-e40ce0597ac9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70a7ae14-0130-4733-ae4b-e40ce0597ac9-maas-api-xks-ca\") pod \"70a7ae14-0130-4733-ae4b-e40ce0597ac9\" (UID: \"70a7ae14-0130-4733-ae4b-e40ce0597ac9\") " Oct 30 21:38:48.061582 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:48.061490 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70a7ae14-0130-4733-ae4b-e40ce0597ac9-kube-api-access-n4chp\" (UniqueName: \"kubernetes.io/projected/70a7ae14-0130-4733-ae4b-e40ce0597ac9-kube-api-access-n4chp\") pod \"70a7ae14-0130-4733-ae4b-e40ce0597ac9\" (UID: \"70a7ae14-0130-4733-ae4b-e40ce0597ac9\") " Oct 30 21:38:48.061770 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:48.061745 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70a7ae14-0130-4733-ae4b-e40ce0597ac9-maas-api-openshift-ca" pod "70a7ae14-0130-4733-ae4b-e40ce0597ac9" (UID: "70a7ae14-0130-4733-ae4b-e40ce0597ac9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:38:48.063588 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:48.063557 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70a7ae14-0130-4733-ae4b-e40ce0597ac9-maas-api-xks-ca" pod "70a7ae14-0130-4733-ae4b-e40ce0597ac9" (UID: "70a7ae14-0130-4733-ae4b-e40ce0597ac9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:38:48.063699 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:48.063634 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70a7ae14-0130-4733-ae4b-e40ce0597ac9-kube-api-access-n4chp" pod "70a7ae14-0130-4733-ae4b-e40ce0597ac9" (UID: "70a7ae14-0130-4733-ae4b-e40ce0597ac9"). InnerVolumeSpecName "kube-api-access-n4chp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:38:48.163147 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:48.163119 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70a7ae14-0130-4733-ae4b-e40ce0597ac9-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:48.163147 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:48.163145 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4chp\" (UniqueName: \"kubernetes.io/projected/70a7ae14-0130-4733-ae4b-e40ce0597ac9-kube-api-access-n4chp\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:48.163147 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:48.163155 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70a7ae14-0130-4733-ae4b-e40ce0597ac9-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:48.793242 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:48.793145 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef54ebcb-29n9k" event={"ID":"70a7ae14-0130-4733-ae4b-e40ce0597ac9","Type":"ContainerDied","Data":"cafddd124ea6334b40310a390670fb109ef2add131d1e558cea1252a29fd4196"} Oct 30 21:38:48.793242 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:48.793168 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef54ebcb-29n9k" Oct 30 21:38:48.793242 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:48.793196 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cafddd124ea6334b40310a390670fb109ef2add131d1e558cea1252a29fd4196" Oct 30 21:38:58.653598 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:58.653561 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d13b782e-2zxh4"] Oct 30 21:38:58.654159 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:58.654120 5748 cpu_manager.go:414] "containerMap: removing container" podUID="38111ab1-6d6e-4392-8fd6-8fa44bbdf851" containerName="revoke-keys" Oct 30 21:38:58.654159 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:58.654139 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38111ab1-6d6e-4392-8fd6-8fa44bbdf851" containerName="revoke-keys" Oct 30 21:38:58.654159 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:58.654155 5748 cpu_manager.go:414] "containerMap: removing container" podUID="70a7ae14-0130-4733-ae4b-e40ce0597ac9" containerName="revoke-keys" Oct 30 21:38:58.654267 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:58.654163 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70a7ae14-0130-4733-ae4b-e40ce0597ac9" containerName="revoke-keys" Oct 30 21:38:58.654267 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:58.654181 5748 cpu_manager.go:414] "containerMap: removing container" podUID="cba56321-fa3c-4c0e-a7d7-7f89d9dcf866" containerName="revoke-keys" Oct 30 21:38:58.654267 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:58.654189 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cba56321-fa3c-4c0e-a7d7-7f89d9dcf866" containerName="revoke-keys" Oct 30 21:38:58.654378 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:58.654363 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="38111ab1-6d6e-4392-8fd6-8fa44bbdf851" containerName="revoke-keys" Oct 30 21:38:58.654419 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:58.654381 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="70a7ae14-0130-4733-ae4b-e40ce0597ac9" containerName="revoke-keys" Oct 30 21:38:58.654419 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:58.654391 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="cba56321-fa3c-4c0e-a7d7-7f89d9dcf866" containerName="revoke-keys" Oct 30 21:38:58.660534 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:58.660511 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13b782e-2zxh4" Oct 30 21:38:58.715538 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:58.715499 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d13b782e-2zxh4"] Oct 30 21:38:58.763574 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:58.763541 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:38:58.763574 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:58.763566 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:38:58.763800 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:58.763618 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gld4\" (UniqueName: \"kubernetes.io/projected/82a501af-15ab-4fd6-b6bf-38e89b5c1727-kube-api-access-2gld4\") pod \"maas-api-revoke-sub-d13b782e-2zxh4\" (UID: \"82a501af-15ab-4fd6-b6bf-38e89b5c1727\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13b782e-2zxh4" Oct 30 21:38:58.763800 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:58.763624 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:38:58.763800 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:58.763715 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82a501af-15ab-4fd6-b6bf-38e89b5c1727-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d13b782e-2zxh4\" (UID: \"82a501af-15ab-4fd6-b6bf-38e89b5c1727\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13b782e-2zxh4" Oct 30 21:38:58.763932 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:58.763855 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82a501af-15ab-4fd6-b6bf-38e89b5c1727-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d13b782e-2zxh4\" (UID: \"82a501af-15ab-4fd6-b6bf-38e89b5c1727\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13b782e-2zxh4" Oct 30 21:38:58.764373 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:58.764359 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 21:38:58.864335 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:58.864304 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gld4\" (UniqueName: \"kubernetes.io/projected/82a501af-15ab-4fd6-b6bf-38e89b5c1727-kube-api-access-2gld4\") pod \"maas-api-revoke-sub-d13b782e-2zxh4\" (UID: \"82a501af-15ab-4fd6-b6bf-38e89b5c1727\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13b782e-2zxh4" Oct 30 21:38:58.864532 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:58.864350 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82a501af-15ab-4fd6-b6bf-38e89b5c1727-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d13b782e-2zxh4\" (UID: \"82a501af-15ab-4fd6-b6bf-38e89b5c1727\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13b782e-2zxh4" Oct 30 21:38:58.864532 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:58.864432 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82a501af-15ab-4fd6-b6bf-38e89b5c1727-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d13b782e-2zxh4\" (UID: \"82a501af-15ab-4fd6-b6bf-38e89b5c1727\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13b782e-2zxh4" Oct 30 21:38:58.865061 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:58.865042 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82a501af-15ab-4fd6-b6bf-38e89b5c1727-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d13b782e-2zxh4\" (UID: \"82a501af-15ab-4fd6-b6bf-38e89b5c1727\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13b782e-2zxh4" Oct 30 21:38:58.866736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:58.866714 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82a501af-15ab-4fd6-b6bf-38e89b5c1727-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d13b782e-2zxh4\" (UID: \"82a501af-15ab-4fd6-b6bf-38e89b5c1727\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13b782e-2zxh4" Oct 30 21:38:58.925807 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:58.925738 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gld4\" (UniqueName: \"kubernetes.io/projected/82a501af-15ab-4fd6-b6bf-38e89b5c1727-kube-api-access-2gld4\") pod \"maas-api-revoke-sub-d13b782e-2zxh4\" (UID: \"82a501af-15ab-4fd6-b6bf-38e89b5c1727\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13b782e-2zxh4" Oct 30 21:38:58.974225 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:58.974194 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13b782e-2zxh4" Oct 30 21:38:59.354852 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:59.354822 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d13b782e-2zxh4"] Oct 30 21:38:59.356890 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:38:59.356863 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82a501af_15ab_4fd6_b6bf_38e89b5c1727.slice/crio-796e9d066786a35b2c978872894a41189af233a0d74ed45874b6aab4ab13e158 WatchSource:0}: Error finding container 796e9d066786a35b2c978872894a41189af233a0d74ed45874b6aab4ab13e158: Status 404 returned error can't find the container with id 796e9d066786a35b2c978872894a41189af233a0d74ed45874b6aab4ab13e158 Oct 30 21:38:59.853593 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:59.853561 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13b782e-2zxh4" event={"ID":"82a501af-15ab-4fd6-b6bf-38e89b5c1727","Type":"ContainerStarted","Data":"22234416d4bdfa4ff363df77eadb28a24a61cb66a8aa6cfd3b77f4bb98203c2c"} Oct 30 21:38:59.853997 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:59.853611 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13b782e-2zxh4" event={"ID":"82a501af-15ab-4fd6-b6bf-38e89b5c1727","Type":"ContainerStarted","Data":"796e9d066786a35b2c978872894a41189af233a0d74ed45874b6aab4ab13e158"} Oct 30 21:38:59.980186 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:38:59.980130 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13b782e-2zxh4" podStartSLOduration=1.9801165969999999 podStartE2EDuration="1.980116597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:59.978101502 +0000 UTC m=+923.720622449" watchObservedRunningTime="2026-10-30 21:38:59.980116597 +0000 UTC m=+923.722637532" Oct 30 21:39:00.148574 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:39:00.148529 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:39:00.287193 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:00.287108 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62098d29-cggjg"] Oct 30 21:39:00.293352 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:00.293318 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62098d29-cggjg" Oct 30 21:39:00.349716 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:00.349674 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62098d29-cggjg"] Oct 30 21:39:00.378447 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:00.378414 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c463f776-084f-43de-98f1-1035dcc8d43a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62098d29-cggjg\" (UID: \"c463f776-084f-43de-98f1-1035dcc8d43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62098d29-cggjg" Oct 30 21:39:00.378627 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:00.378470 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngc5t\" (UniqueName: \"kubernetes.io/projected/c463f776-084f-43de-98f1-1035dcc8d43a-kube-api-access-ngc5t\") pod \"maas-api-revoke-sub-62098d29-cggjg\" (UID: \"c463f776-084f-43de-98f1-1035dcc8d43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62098d29-cggjg" Oct 30 21:39:00.378627 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:00.378590 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c463f776-084f-43de-98f1-1035dcc8d43a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62098d29-cggjg\" (UID: \"c463f776-084f-43de-98f1-1035dcc8d43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62098d29-cggjg" Oct 30 21:39:00.479954 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:00.479910 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngc5t\" (UniqueName: \"kubernetes.io/projected/c463f776-084f-43de-98f1-1035dcc8d43a-kube-api-access-ngc5t\") pod \"maas-api-revoke-sub-62098d29-cggjg\" (UID: \"c463f776-084f-43de-98f1-1035dcc8d43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62098d29-cggjg" Oct 30 21:39:00.480134 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:00.480023 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c463f776-084f-43de-98f1-1035dcc8d43a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62098d29-cggjg\" (UID: \"c463f776-084f-43de-98f1-1035dcc8d43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62098d29-cggjg" Oct 30 21:39:00.480479 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:00.480453 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c463f776-084f-43de-98f1-1035dcc8d43a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62098d29-cggjg\" (UID: \"c463f776-084f-43de-98f1-1035dcc8d43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62098d29-cggjg" Oct 30 21:39:00.480781 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:00.480754 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c463f776-084f-43de-98f1-1035dcc8d43a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62098d29-cggjg\" (UID: \"c463f776-084f-43de-98f1-1035dcc8d43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62098d29-cggjg" Oct 30 21:39:00.482667 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:00.482646 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c463f776-084f-43de-98f1-1035dcc8d43a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62098d29-cggjg\" (UID: \"c463f776-084f-43de-98f1-1035dcc8d43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62098d29-cggjg" Oct 30 21:39:00.541773 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:00.541690 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngc5t\" (UniqueName: \"kubernetes.io/projected/c463f776-084f-43de-98f1-1035dcc8d43a-kube-api-access-ngc5t\") pod \"maas-api-revoke-sub-62098d29-cggjg\" (UID: \"c463f776-084f-43de-98f1-1035dcc8d43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62098d29-cggjg" Oct 30 21:39:00.607638 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:00.607599 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62098d29-cggjg" Oct 30 21:39:00.860363 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:00.860277 5748 generic.go:356] "Generic (PLEG): container finished" podID="82a501af-15ab-4fd6-b6bf-38e89b5c1727" containerID="22234416d4bdfa4ff363df77eadb28a24a61cb66a8aa6cfd3b77f4bb98203c2c" exitCode=0 Oct 30 21:39:00.860732 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:00.860365 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13b782e-2zxh4" event={"ID":"82a501af-15ab-4fd6-b6bf-38e89b5c1727","Type":"ContainerDied","Data":"22234416d4bdfa4ff363df77eadb28a24a61cb66a8aa6cfd3b77f4bb98203c2c"} Oct 30 21:39:00.985300 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:39:00.985263 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc463f776_084f_43de_98f1_1035dcc8d43a.slice/crio-bfb03641beb5e13c888de3fce017676a703201a281fa9f8c6066a0aaa502d7d4 WatchSource:0}: Error finding container bfb03641beb5e13c888de3fce017676a703201a281fa9f8c6066a0aaa502d7d4: Status 404 returned error can't find the container with id bfb03641beb5e13c888de3fce017676a703201a281fa9f8c6066a0aaa502d7d4 Oct 30 21:39:01.029757 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:01.029723 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62098d29-cggjg"] Oct 30 21:39:01.868360 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:01.868320 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62098d29-cggjg" event={"ID":"c463f776-084f-43de-98f1-1035dcc8d43a","Type":"ContainerStarted","Data":"0ac75adabb16aa8c20f930ed700b0f92a280a61275033f0f7a8b7168f92fb66f"} Oct 30 21:39:01.868782 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:01.868385 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62098d29-cggjg" event={"ID":"c463f776-084f-43de-98f1-1035dcc8d43a","Type":"ContainerStarted","Data":"bfb03641beb5e13c888de3fce017676a703201a281fa9f8c6066a0aaa502d7d4"} Oct 30 21:39:02.013587 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:02.013560 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13b782e-2zxh4" Oct 30 21:39:02.099226 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:02.099186 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82a501af-15ab-4fd6-b6bf-38e89b5c1727-kube-api-access-2gld4\" (UniqueName: \"kubernetes.io/projected/82a501af-15ab-4fd6-b6bf-38e89b5c1727-kube-api-access-2gld4\") pod \"82a501af-15ab-4fd6-b6bf-38e89b5c1727\" (UID: \"82a501af-15ab-4fd6-b6bf-38e89b5c1727\") " Oct 30 21:39:02.099405 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:02.099327 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82a501af-15ab-4fd6-b6bf-38e89b5c1727-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82a501af-15ab-4fd6-b6bf-38e89b5c1727-maas-api-xks-ca\") pod \"82a501af-15ab-4fd6-b6bf-38e89b5c1727\" (UID: \"82a501af-15ab-4fd6-b6bf-38e89b5c1727\") " Oct 30 21:39:02.099405 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:02.099377 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82a501af-15ab-4fd6-b6bf-38e89b5c1727-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82a501af-15ab-4fd6-b6bf-38e89b5c1727-maas-api-openshift-ca\") pod \"82a501af-15ab-4fd6-b6bf-38e89b5c1727\" (UID: \"82a501af-15ab-4fd6-b6bf-38e89b5c1727\") " Oct 30 21:39:02.099796 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:02.099770 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82a501af-15ab-4fd6-b6bf-38e89b5c1727-maas-api-openshift-ca" pod "82a501af-15ab-4fd6-b6bf-38e89b5c1727" (UID: "82a501af-15ab-4fd6-b6bf-38e89b5c1727"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:39:02.101359 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:02.101335 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82a501af-15ab-4fd6-b6bf-38e89b5c1727-kube-api-access-2gld4" pod "82a501af-15ab-4fd6-b6bf-38e89b5c1727" (UID: "82a501af-15ab-4fd6-b6bf-38e89b5c1727"). InnerVolumeSpecName "kube-api-access-2gld4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:39:02.101441 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:02.101343 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82a501af-15ab-4fd6-b6bf-38e89b5c1727-maas-api-xks-ca" pod "82a501af-15ab-4fd6-b6bf-38e89b5c1727" (UID: "82a501af-15ab-4fd6-b6bf-38e89b5c1727"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:39:02.133357 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:02.133266 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62098d29-cggjg" podStartSLOduration=2.133247118 podStartE2EDuration="2.133247118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:01.989218515 +0000 UTC m=+925.731739450" watchObservedRunningTime="2026-10-30 21:39:02.133247118 +0000 UTC m=+925.875768053" Oct 30 21:39:02.201304 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:02.201266 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82a501af-15ab-4fd6-b6bf-38e89b5c1727-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:02.201304 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:02.201298 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82a501af-15ab-4fd6-b6bf-38e89b5c1727-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:02.201304 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:02.201307 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gld4\" (UniqueName: \"kubernetes.io/projected/82a501af-15ab-4fd6-b6bf-38e89b5c1727-kube-api-access-2gld4\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:02.875647 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:02.875561 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13b782e-2zxh4" Oct 30 21:39:02.875647 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:02.875576 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13b782e-2zxh4" event={"ID":"82a501af-15ab-4fd6-b6bf-38e89b5c1727","Type":"ContainerDied","Data":"796e9d066786a35b2c978872894a41189af233a0d74ed45874b6aab4ab13e158"} Oct 30 21:39:02.875647 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:02.875641 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="796e9d066786a35b2c978872894a41189af233a0d74ed45874b6aab4ab13e158" Oct 30 21:39:02.877276 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:02.877251 5748 generic.go:356] "Generic (PLEG): container finished" podID="c463f776-084f-43de-98f1-1035dcc8d43a" containerID="0ac75adabb16aa8c20f930ed700b0f92a280a61275033f0f7a8b7168f92fb66f" exitCode=0 Oct 30 21:39:02.877426 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:02.877334 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62098d29-cggjg" event={"ID":"c463f776-084f-43de-98f1-1035dcc8d43a","Type":"ContainerDied","Data":"0ac75adabb16aa8c20f930ed700b0f92a280a61275033f0f7a8b7168f92fb66f"} Oct 30 21:39:03.455002 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:03.454961 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-733e3b40-v6dsk"] Oct 30 21:39:03.455567 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:03.455546 5748 cpu_manager.go:414] "containerMap: removing container" podUID="82a501af-15ab-4fd6-b6bf-38e89b5c1727" containerName="revoke-keys" Oct 30 21:39:03.455665 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:03.455569 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82a501af-15ab-4fd6-b6bf-38e89b5c1727" containerName="revoke-keys" Oct 30 21:39:03.455719 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:03.455678 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="82a501af-15ab-4fd6-b6bf-38e89b5c1727" containerName="revoke-keys" Oct 30 21:39:03.459095 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:03.459073 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-733e3b40-v6dsk" Oct 30 21:39:03.515205 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:03.515168 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a47cdb8d-3422-4853-b26e-6aa87fedfb4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-733e3b40-v6dsk\" (UID: \"a47cdb8d-3422-4853-b26e-6aa87fedfb4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-733e3b40-v6dsk" Oct 30 21:39:03.515380 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:03.515227 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a47cdb8d-3422-4853-b26e-6aa87fedfb4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-733e3b40-v6dsk\" (UID: \"a47cdb8d-3422-4853-b26e-6aa87fedfb4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-733e3b40-v6dsk" Oct 30 21:39:03.515435 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:03.515409 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzw7b\" (UniqueName: \"kubernetes.io/projected/a47cdb8d-3422-4853-b26e-6aa87fedfb4b-kube-api-access-zzw7b\") pod \"maas-api-revoke-sub-733e3b40-v6dsk\" (UID: \"a47cdb8d-3422-4853-b26e-6aa87fedfb4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-733e3b40-v6dsk" Oct 30 21:39:03.519210 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:03.519182 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-733e3b40-v6dsk"] Oct 30 21:39:03.616124 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:03.616086 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzw7b\" (UniqueName: \"kubernetes.io/projected/a47cdb8d-3422-4853-b26e-6aa87fedfb4b-kube-api-access-zzw7b\") pod \"maas-api-revoke-sub-733e3b40-v6dsk\" (UID: \"a47cdb8d-3422-4853-b26e-6aa87fedfb4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-733e3b40-v6dsk" Oct 30 21:39:03.616299 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:03.616152 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a47cdb8d-3422-4853-b26e-6aa87fedfb4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-733e3b40-v6dsk\" (UID: \"a47cdb8d-3422-4853-b26e-6aa87fedfb4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-733e3b40-v6dsk" Oct 30 21:39:03.616299 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:03.616193 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a47cdb8d-3422-4853-b26e-6aa87fedfb4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-733e3b40-v6dsk\" (UID: \"a47cdb8d-3422-4853-b26e-6aa87fedfb4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-733e3b40-v6dsk" Oct 30 21:39:03.616949 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:03.616920 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a47cdb8d-3422-4853-b26e-6aa87fedfb4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-733e3b40-v6dsk\" (UID: \"a47cdb8d-3422-4853-b26e-6aa87fedfb4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-733e3b40-v6dsk" Oct 30 21:39:03.618626 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:03.618605 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a47cdb8d-3422-4853-b26e-6aa87fedfb4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-733e3b40-v6dsk\" (UID: \"a47cdb8d-3422-4853-b26e-6aa87fedfb4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-733e3b40-v6dsk" Oct 30 21:39:03.678613 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:03.678574 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzw7b\" (UniqueName: \"kubernetes.io/projected/a47cdb8d-3422-4853-b26e-6aa87fedfb4b-kube-api-access-zzw7b\") pod \"maas-api-revoke-sub-733e3b40-v6dsk\" (UID: \"a47cdb8d-3422-4853-b26e-6aa87fedfb4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-733e3b40-v6dsk" Oct 30 21:39:03.772095 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:03.772007 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-733e3b40-v6dsk" Oct 30 21:39:03.843968 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:03.843927 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9"] Oct 30 21:39:03.849316 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:03.849287 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9" Oct 30 21:39:03.905790 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:03.905742 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-1de75a-openshift-default-dockercfg-rtvkc\"" Oct 30 21:39:03.908964 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:03.908941 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9"] Oct 30 21:39:03.918976 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:03.918938 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26dc7eb4-1548-433d-b7dd-0368d36a466b-istio-podinfo\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9\" (UID: \"26dc7eb4-1548-433d-b7dd-0368d36a466b\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9" Oct 30 21:39:03.919146 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:03.919010 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtr2f\" (UniqueName: \"kubernetes.io/projected/26dc7eb4-1548-433d-b7dd-0368d36a466b-kube-api-access-qtr2f\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9\" (UID: \"26dc7eb4-1548-433d-b7dd-0368d36a466b\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9" Oct 30 21:39:03.919146 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:03.919051 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26dc7eb4-1548-433d-b7dd-0368d36a466b-workload-socket\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9\" (UID: \"26dc7eb4-1548-433d-b7dd-0368d36a466b\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9" Oct 30 21:39:03.919146 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:03.919090 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26dc7eb4-1548-433d-b7dd-0368d36a466b-istiod-ca-cert\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9\" (UID: \"26dc7eb4-1548-433d-b7dd-0368d36a466b\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9" Oct 30 21:39:03.919146 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:03.919129 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26dc7eb4-1548-433d-b7dd-0368d36a466b-credential-socket\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9\" (UID: \"26dc7eb4-1548-433d-b7dd-0368d36a466b\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9" Oct 30 21:39:03.919146 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:03.919143 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26dc7eb4-1548-433d-b7dd-0368d36a466b-workload-certs\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9\" (UID: \"26dc7eb4-1548-433d-b7dd-0368d36a466b\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9" Oct 30 21:39:03.919480 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:03.919210 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26dc7eb4-1548-433d-b7dd-0368d36a466b-istio-data\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9\" (UID: \"26dc7eb4-1548-433d-b7dd-0368d36a466b\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9" Oct 30 21:39:03.919480 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:03.919239 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26dc7eb4-1548-433d-b7dd-0368d36a466b-istio-envoy\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9\" (UID: \"26dc7eb4-1548-433d-b7dd-0368d36a466b\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9" Oct 30 21:39:03.919480 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:03.919257 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26dc7eb4-1548-433d-b7dd-0368d36a466b-istio-token\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9\" (UID: \"26dc7eb4-1548-433d-b7dd-0368d36a466b\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9" Oct 30 21:39:04.019933 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:04.019879 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26dc7eb4-1548-433d-b7dd-0368d36a466b-istio-podinfo\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9\" (UID: \"26dc7eb4-1548-433d-b7dd-0368d36a466b\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9" Oct 30 21:39:04.020094 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:04.020042 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtr2f\" (UniqueName: \"kubernetes.io/projected/26dc7eb4-1548-433d-b7dd-0368d36a466b-kube-api-access-qtr2f\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9\" (UID: \"26dc7eb4-1548-433d-b7dd-0368d36a466b\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9" Oct 30 21:39:04.020094 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:04.020083 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26dc7eb4-1548-433d-b7dd-0368d36a466b-workload-socket\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9\" (UID: \"26dc7eb4-1548-433d-b7dd-0368d36a466b\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9" Oct 30 21:39:04.020186 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:04.020125 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26dc7eb4-1548-433d-b7dd-0368d36a466b-istiod-ca-cert\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9\" (UID: \"26dc7eb4-1548-433d-b7dd-0368d36a466b\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9" Oct 30 21:39:04.020306 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:04.020266 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26dc7eb4-1548-433d-b7dd-0368d36a466b-credential-socket\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9\" (UID: \"26dc7eb4-1548-433d-b7dd-0368d36a466b\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9" Oct 30 21:39:04.020392 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:04.020323 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26dc7eb4-1548-433d-b7dd-0368d36a466b-workload-certs\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9\" (UID: \"26dc7eb4-1548-433d-b7dd-0368d36a466b\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9" Oct 30 21:39:04.020457 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:04.020426 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26dc7eb4-1548-433d-b7dd-0368d36a466b-istio-data\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9\" (UID: \"26dc7eb4-1548-433d-b7dd-0368d36a466b\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9" Oct 30 21:39:04.020510 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:04.020475 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26dc7eb4-1548-433d-b7dd-0368d36a466b-istio-envoy\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9\" (UID: \"26dc7eb4-1548-433d-b7dd-0368d36a466b\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9" Oct 30 21:39:04.020510 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:04.020502 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26dc7eb4-1548-433d-b7dd-0368d36a466b-istio-token\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9\" (UID: \"26dc7eb4-1548-433d-b7dd-0368d36a466b\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9" Oct 30 21:39:04.020610 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:04.020540 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26dc7eb4-1548-433d-b7dd-0368d36a466b-workload-socket\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9\" (UID: \"26dc7eb4-1548-433d-b7dd-0368d36a466b\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9" Oct 30 21:39:04.020975 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:04.020668 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26dc7eb4-1548-433d-b7dd-0368d36a466b-credential-socket\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9\" (UID: \"26dc7eb4-1548-433d-b7dd-0368d36a466b\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9" Oct 30 21:39:04.020975 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:04.020831 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26dc7eb4-1548-433d-b7dd-0368d36a466b-workload-certs\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9\" (UID: \"26dc7eb4-1548-433d-b7dd-0368d36a466b\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9" Oct 30 21:39:04.020975 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:04.020936 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26dc7eb4-1548-433d-b7dd-0368d36a466b-istiod-ca-cert\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9\" (UID: \"26dc7eb4-1548-433d-b7dd-0368d36a466b\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9" Oct 30 21:39:04.021169 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:04.020988 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26dc7eb4-1548-433d-b7dd-0368d36a466b-istio-data\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9\" (UID: \"26dc7eb4-1548-433d-b7dd-0368d36a466b\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9" Oct 30 21:39:04.022736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:04.022674 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26dc7eb4-1548-433d-b7dd-0368d36a466b-istio-envoy\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9\" (UID: \"26dc7eb4-1548-433d-b7dd-0368d36a466b\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9" Oct 30 21:39:04.022865 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:04.022846 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26dc7eb4-1548-433d-b7dd-0368d36a466b-istio-podinfo\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9\" (UID: \"26dc7eb4-1548-433d-b7dd-0368d36a466b\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9" Oct 30 21:39:04.042838 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:04.042817 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62098d29-cggjg" Oct 30 21:39:04.082466 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:04.082436 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26dc7eb4-1548-433d-b7dd-0368d36a466b-istio-token\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9\" (UID: \"26dc7eb4-1548-433d-b7dd-0368d36a466b\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9" Oct 30 21:39:04.082641 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:04.082616 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtr2f\" (UniqueName: \"kubernetes.io/projected/26dc7eb4-1548-433d-b7dd-0368d36a466b-kube-api-access-qtr2f\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9\" (UID: \"26dc7eb4-1548-433d-b7dd-0368d36a466b\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9" Oct 30 21:39:04.121391 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:04.121356 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c463f776-084f-43de-98f1-1035dcc8d43a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c463f776-084f-43de-98f1-1035dcc8d43a-maas-api-xks-ca\") pod \"c463f776-084f-43de-98f1-1035dcc8d43a\" (UID: \"c463f776-084f-43de-98f1-1035dcc8d43a\") " Oct 30 21:39:04.121554 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:04.121438 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c463f776-084f-43de-98f1-1035dcc8d43a-kube-api-access-ngc5t\" (UniqueName: \"kubernetes.io/projected/c463f776-084f-43de-98f1-1035dcc8d43a-kube-api-access-ngc5t\") pod \"c463f776-084f-43de-98f1-1035dcc8d43a\" (UID: \"c463f776-084f-43de-98f1-1035dcc8d43a\") " Oct 30 21:39:04.121610 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:04.121594 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c463f776-084f-43de-98f1-1035dcc8d43a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c463f776-084f-43de-98f1-1035dcc8d43a-maas-api-openshift-ca\") pod \"c463f776-084f-43de-98f1-1035dcc8d43a\" (UID: \"c463f776-084f-43de-98f1-1035dcc8d43a\") " Oct 30 21:39:04.121954 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:04.121927 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c463f776-084f-43de-98f1-1035dcc8d43a-maas-api-openshift-ca" pod "c463f776-084f-43de-98f1-1035dcc8d43a" (UID: "c463f776-084f-43de-98f1-1035dcc8d43a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:39:04.123478 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:04.123452 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c463f776-084f-43de-98f1-1035dcc8d43a-maas-api-xks-ca" pod "c463f776-084f-43de-98f1-1035dcc8d43a" (UID: "c463f776-084f-43de-98f1-1035dcc8d43a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:39:04.123478 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:04.123462 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c463f776-084f-43de-98f1-1035dcc8d43a-kube-api-access-ngc5t" pod "c463f776-084f-43de-98f1-1035dcc8d43a" (UID: "c463f776-084f-43de-98f1-1035dcc8d43a"). InnerVolumeSpecName "kube-api-access-ngc5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:39:04.147857 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:04.147834 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-733e3b40-v6dsk"] Oct 30 21:39:04.149410 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:39:04.149383 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda47cdb8d_3422_4853_b26e_6aa87fedfb4b.slice/crio-dbbb356378ebc6c0ea027039df6fc4af34f34d8201865b94819c624933037500 WatchSource:0}: Error finding container dbbb356378ebc6c0ea027039df6fc4af34f34d8201865b94819c624933037500: Status 404 returned error can't find the container with id dbbb356378ebc6c0ea027039df6fc4af34f34d8201865b94819c624933037500 Oct 30 21:39:04.165934 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:04.165573 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9" Oct 30 21:39:04.223018 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:04.222992 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngc5t\" (UniqueName: \"kubernetes.io/projected/c463f776-084f-43de-98f1-1035dcc8d43a-kube-api-access-ngc5t\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:04.223131 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:04.223020 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c463f776-084f-43de-98f1-1035dcc8d43a-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:04.223131 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:04.223032 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c463f776-084f-43de-98f1-1035dcc8d43a-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:04.599713 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:04.599687 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9"] Oct 30 21:39:04.601798 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:39:04.601769 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26dc7eb4_1548_433d_b7dd_0368d36a466b.slice/crio-822ad153718a74a8cb7d365018ff7017e8c71c96191f306875f8615df507d05d WatchSource:0}: Error finding container 822ad153718a74a8cb7d365018ff7017e8c71c96191f306875f8615df507d05d: Status 404 returned error can't find the container with id 822ad153718a74a8cb7d365018ff7017e8c71c96191f306875f8615df507d05d Oct 30 21:39:04.604257 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:04.604222 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:39:04.604362 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:04.604302 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:39:04.604362 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:04.604350 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:39:04.903436 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:04.903395 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9" event={"ID":"26dc7eb4-1548-433d-b7dd-0368d36a466b","Type":"ContainerStarted","Data":"5b2ac266fc4a7d605d6f354af957294d12a393de1333396aef074274c65fded5"} Oct 30 21:39:04.903641 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:04.903465 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9" event={"ID":"26dc7eb4-1548-433d-b7dd-0368d36a466b","Type":"ContainerStarted","Data":"822ad153718a74a8cb7d365018ff7017e8c71c96191f306875f8615df507d05d"} Oct 30 21:39:04.905364 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:04.905331 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-733e3b40-v6dsk" event={"ID":"a47cdb8d-3422-4853-b26e-6aa87fedfb4b","Type":"ContainerStarted","Data":"86f7d7e6a2002c1c763b509faac025d1e9739b75c583f81ab363d3318fceea47"} Oct 30 21:39:04.905495 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:04.905380 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-733e3b40-v6dsk" event={"ID":"a47cdb8d-3422-4853-b26e-6aa87fedfb4b","Type":"ContainerStarted","Data":"dbbb356378ebc6c0ea027039df6fc4af34f34d8201865b94819c624933037500"} Oct 30 21:39:04.906868 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:04.906840 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62098d29-cggjg" event={"ID":"c463f776-084f-43de-98f1-1035dcc8d43a","Type":"ContainerDied","Data":"bfb03641beb5e13c888de3fce017676a703201a281fa9f8c6066a0aaa502d7d4"} Oct 30 21:39:04.907304 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:04.906889 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bfb03641beb5e13c888de3fce017676a703201a281fa9f8c6066a0aaa502d7d4" Oct 30 21:39:04.907304 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:04.906943 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62098d29-cggjg" Oct 30 21:39:05.029218 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:05.029156 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9" podStartSLOduration=2.02913576 podStartE2EDuration="2.02913576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:05.027972848 +0000 UTC m=+928.770493780" watchObservedRunningTime="2026-10-30 21:39:05.02913576 +0000 UTC m=+928.771656695" Oct 30 21:39:05.166308 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:05.166216 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9" Oct 30 21:39:05.167820 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:05.167796 5748 patch_prober.go:29] interesting pod/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 21:39:05.167887 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:05.167850 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9" podUID="26dc7eb4-1548-433d-b7dd-0368d36a466b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 21:39:05.208918 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:05.206344 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-733e3b40-v6dsk" podStartSLOduration=2.20632541 podStartE2EDuration="2.20632541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:05.206153543 +0000 UTC m=+928.948674471" watchObservedRunningTime="2026-10-30 21:39:05.20632541 +0000 UTC m=+928.948846338" Oct 30 21:39:05.930848 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:05.930814 5748 generic.go:356] "Generic (PLEG): container finished" podID="a47cdb8d-3422-4853-b26e-6aa87fedfb4b" containerID="86f7d7e6a2002c1c763b509faac025d1e9739b75c583f81ab363d3318fceea47" exitCode=0 Oct 30 21:39:05.931280 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:05.930874 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-733e3b40-v6dsk" event={"ID":"a47cdb8d-3422-4853-b26e-6aa87fedfb4b","Type":"ContainerDied","Data":"86f7d7e6a2002c1c763b509faac025d1e9739b75c583f81ab363d3318fceea47"} Oct 30 21:39:06.166493 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:06.166455 5748 patch_prober.go:29] interesting pod/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 21:39:06.166679 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:06.166525 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9" podUID="26dc7eb4-1548-433d-b7dd-0368d36a466b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 21:39:06.738263 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:06.738226 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09ae27cf-jqpxr"] Oct 30 21:39:06.738747 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:06.738731 5748 cpu_manager.go:414] "containerMap: removing container" podUID="c463f776-084f-43de-98f1-1035dcc8d43a" containerName="revoke-keys" Oct 30 21:39:06.738812 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:06.738748 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c463f776-084f-43de-98f1-1035dcc8d43a" containerName="revoke-keys" Oct 30 21:39:06.738849 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:06.738834 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="c463f776-084f-43de-98f1-1035dcc8d43a" containerName="revoke-keys" Oct 30 21:39:06.742058 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:06.742041 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09ae27cf-jqpxr" Oct 30 21:39:06.831849 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:06.830571 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09ae27cf-jqpxr"] Oct 30 21:39:06.857021 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:06.856975 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qmkj\" (UniqueName: \"kubernetes.io/projected/1597353f-0050-4a88-89df-ccec34a0e2b0-kube-api-access-9qmkj\") pod \"maas-api-revoke-sub-09ae27cf-jqpxr\" (UID: \"1597353f-0050-4a88-89df-ccec34a0e2b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09ae27cf-jqpxr" Oct 30 21:39:06.857021 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:06.857019 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1597353f-0050-4a88-89df-ccec34a0e2b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09ae27cf-jqpxr\" (UID: \"1597353f-0050-4a88-89df-ccec34a0e2b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09ae27cf-jqpxr" Oct 30 21:39:06.857274 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:06.857159 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1597353f-0050-4a88-89df-ccec34a0e2b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09ae27cf-jqpxr\" (UID: \"1597353f-0050-4a88-89df-ccec34a0e2b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09ae27cf-jqpxr" Oct 30 21:39:06.959005 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:06.958964 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qmkj\" (UniqueName: \"kubernetes.io/projected/1597353f-0050-4a88-89df-ccec34a0e2b0-kube-api-access-9qmkj\") pod \"maas-api-revoke-sub-09ae27cf-jqpxr\" (UID: \"1597353f-0050-4a88-89df-ccec34a0e2b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09ae27cf-jqpxr" Oct 30 21:39:06.959477 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:06.959029 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1597353f-0050-4a88-89df-ccec34a0e2b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09ae27cf-jqpxr\" (UID: \"1597353f-0050-4a88-89df-ccec34a0e2b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09ae27cf-jqpxr" Oct 30 21:39:06.959477 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:06.959146 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1597353f-0050-4a88-89df-ccec34a0e2b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09ae27cf-jqpxr\" (UID: \"1597353f-0050-4a88-89df-ccec34a0e2b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09ae27cf-jqpxr" Oct 30 21:39:06.959477 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:06.959396 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d96d4216-9zkrg"] Oct 30 21:39:06.959880 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:06.959851 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1597353f-0050-4a88-89df-ccec34a0e2b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09ae27cf-jqpxr\" (UID: \"1597353f-0050-4a88-89df-ccec34a0e2b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09ae27cf-jqpxr" Oct 30 21:39:06.961590 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:06.961566 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1597353f-0050-4a88-89df-ccec34a0e2b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09ae27cf-jqpxr\" (UID: \"1597353f-0050-4a88-89df-ccec34a0e2b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09ae27cf-jqpxr" Oct 30 21:39:06.963444 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:06.963425 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96d4216-9zkrg" Oct 30 21:39:07.038034 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:07.038006 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d96d4216-9zkrg"] Oct 30 21:39:07.042834 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:07.042810 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qmkj\" (UniqueName: \"kubernetes.io/projected/1597353f-0050-4a88-89df-ccec34a0e2b0-kube-api-access-9qmkj\") pod \"maas-api-revoke-sub-09ae27cf-jqpxr\" (UID: \"1597353f-0050-4a88-89df-ccec34a0e2b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09ae27cf-jqpxr" Oct 30 21:39:07.055506 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:07.055171 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09ae27cf-jqpxr" Oct 30 21:39:07.060300 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:07.060253 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f2cd81a-d648-4baf-94c7-24ffeadce3d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d96d4216-9zkrg\" (UID: \"2f2cd81a-d648-4baf-94c7-24ffeadce3d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96d4216-9zkrg" Oct 30 21:39:07.060467 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:07.060344 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgz5n\" (UniqueName: \"kubernetes.io/projected/2f2cd81a-d648-4baf-94c7-24ffeadce3d7-kube-api-access-tgz5n\") pod \"maas-api-revoke-sub-d96d4216-9zkrg\" (UID: \"2f2cd81a-d648-4baf-94c7-24ffeadce3d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96d4216-9zkrg" Oct 30 21:39:07.060467 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:07.060410 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f2cd81a-d648-4baf-94c7-24ffeadce3d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d96d4216-9zkrg\" (UID: \"2f2cd81a-d648-4baf-94c7-24ffeadce3d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96d4216-9zkrg" Oct 30 21:39:07.113933 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:07.107183 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-733e3b40-v6dsk" Oct 30 21:39:07.161196 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:07.161166 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a47cdb8d-3422-4853-b26e-6aa87fedfb4b-kube-api-access-zzw7b\" (UniqueName: \"kubernetes.io/projected/a47cdb8d-3422-4853-b26e-6aa87fedfb4b-kube-api-access-zzw7b\") pod \"a47cdb8d-3422-4853-b26e-6aa87fedfb4b\" (UID: \"a47cdb8d-3422-4853-b26e-6aa87fedfb4b\") " Oct 30 21:39:07.161709 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:07.161424 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a47cdb8d-3422-4853-b26e-6aa87fedfb4b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a47cdb8d-3422-4853-b26e-6aa87fedfb4b-maas-api-xks-ca\") pod \"a47cdb8d-3422-4853-b26e-6aa87fedfb4b\" (UID: \"a47cdb8d-3422-4853-b26e-6aa87fedfb4b\") " Oct 30 21:39:07.161823 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:07.161775 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a47cdb8d-3422-4853-b26e-6aa87fedfb4b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a47cdb8d-3422-4853-b26e-6aa87fedfb4b-maas-api-openshift-ca\") pod \"a47cdb8d-3422-4853-b26e-6aa87fedfb4b\" (UID: \"a47cdb8d-3422-4853-b26e-6aa87fedfb4b\") " Oct 30 21:39:07.161982 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:07.161961 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgz5n\" (UniqueName: \"kubernetes.io/projected/2f2cd81a-d648-4baf-94c7-24ffeadce3d7-kube-api-access-tgz5n\") pod \"maas-api-revoke-sub-d96d4216-9zkrg\" (UID: \"2f2cd81a-d648-4baf-94c7-24ffeadce3d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96d4216-9zkrg" Oct 30 21:39:07.162092 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:07.162074 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f2cd81a-d648-4baf-94c7-24ffeadce3d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d96d4216-9zkrg\" (UID: \"2f2cd81a-d648-4baf-94c7-24ffeadce3d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96d4216-9zkrg" Oct 30 21:39:07.162320 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:07.162287 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f2cd81a-d648-4baf-94c7-24ffeadce3d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d96d4216-9zkrg\" (UID: \"2f2cd81a-d648-4baf-94c7-24ffeadce3d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96d4216-9zkrg" Oct 30 21:39:07.163074 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:07.163050 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a47cdb8d-3422-4853-b26e-6aa87fedfb4b-maas-api-openshift-ca" pod "a47cdb8d-3422-4853-b26e-6aa87fedfb4b" (UID: "a47cdb8d-3422-4853-b26e-6aa87fedfb4b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:39:07.163262 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:07.163246 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f2cd81a-d648-4baf-94c7-24ffeadce3d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d96d4216-9zkrg\" (UID: \"2f2cd81a-d648-4baf-94c7-24ffeadce3d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96d4216-9zkrg" Oct 30 21:39:07.164642 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:07.164614 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a47cdb8d-3422-4853-b26e-6aa87fedfb4b-kube-api-access-zzw7b" pod "a47cdb8d-3422-4853-b26e-6aa87fedfb4b" (UID: "a47cdb8d-3422-4853-b26e-6aa87fedfb4b"). InnerVolumeSpecName "kube-api-access-zzw7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:39:07.164755 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:07.164729 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a47cdb8d-3422-4853-b26e-6aa87fedfb4b-maas-api-xks-ca" pod "a47cdb8d-3422-4853-b26e-6aa87fedfb4b" (UID: "a47cdb8d-3422-4853-b26e-6aa87fedfb4b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:39:07.165199 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:07.165179 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f2cd81a-d648-4baf-94c7-24ffeadce3d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d96d4216-9zkrg\" (UID: \"2f2cd81a-d648-4baf-94c7-24ffeadce3d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96d4216-9zkrg" Oct 30 21:39:07.166270 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:07.166252 5748 patch_prober.go:29] interesting pod/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 21:39:07.166366 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:07.166301 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9" podUID="26dc7eb4-1548-433d-b7dd-0368d36a466b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 21:39:07.227751 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:07.227710 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgz5n\" (UniqueName: \"kubernetes.io/projected/2f2cd81a-d648-4baf-94c7-24ffeadce3d7-kube-api-access-tgz5n\") pod \"maas-api-revoke-sub-d96d4216-9zkrg\" (UID: \"2f2cd81a-d648-4baf-94c7-24ffeadce3d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96d4216-9zkrg" Oct 30 21:39:07.263992 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:07.263913 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a47cdb8d-3422-4853-b26e-6aa87fedfb4b-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:07.263992 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:07.263942 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a47cdb8d-3422-4853-b26e-6aa87fedfb4b-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:07.263992 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:07.263952 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzw7b\" (UniqueName: \"kubernetes.io/projected/a47cdb8d-3422-4853-b26e-6aa87fedfb4b-kube-api-access-zzw7b\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:07.287416 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:07.287385 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96d4216-9zkrg" Oct 30 21:39:07.459949 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:07.459923 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09ae27cf-jqpxr"] Oct 30 21:39:07.461668 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:39:07.461640 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1597353f_0050_4a88_89df_ccec34a0e2b0.slice/crio-978f20250008478f213ac31ad02d2c2ec22a4e96cf82d72b59ae629cb7fbd765 WatchSource:0}: Error finding container 978f20250008478f213ac31ad02d2c2ec22a4e96cf82d72b59ae629cb7fbd765: Status 404 returned error can't find the container with id 978f20250008478f213ac31ad02d2c2ec22a4e96cf82d72b59ae629cb7fbd765 Oct 30 21:39:07.687115 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:07.687092 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d96d4216-9zkrg"] Oct 30 21:39:07.689048 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:39:07.689018 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f2cd81a_d648_4baf_94c7_24ffeadce3d7.slice/crio-236eeeaa6dceb1e829908cabf552d55499a203966b5cadf814b7d3e112f68b84 WatchSource:0}: Error finding container 236eeeaa6dceb1e829908cabf552d55499a203966b5cadf814b7d3e112f68b84: Status 404 returned error can't find the container with id 236eeeaa6dceb1e829908cabf552d55499a203966b5cadf814b7d3e112f68b84 Oct 30 21:39:07.942555 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:07.942463 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96d4216-9zkrg" event={"ID":"2f2cd81a-d648-4baf-94c7-24ffeadce3d7","Type":"ContainerStarted","Data":"236eeeaa6dceb1e829908cabf552d55499a203966b5cadf814b7d3e112f68b84"} Oct 30 21:39:07.943772 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:07.943747 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-733e3b40-v6dsk" Oct 30 21:39:07.943917 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:07.943788 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-733e3b40-v6dsk" event={"ID":"a47cdb8d-3422-4853-b26e-6aa87fedfb4b","Type":"ContainerDied","Data":"dbbb356378ebc6c0ea027039df6fc4af34f34d8201865b94819c624933037500"} Oct 30 21:39:07.943917 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:07.943832 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dbbb356378ebc6c0ea027039df6fc4af34f34d8201865b94819c624933037500" Oct 30 21:39:07.945454 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:07.945432 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09ae27cf-jqpxr" event={"ID":"1597353f-0050-4a88-89df-ccec34a0e2b0","Type":"ContainerStarted","Data":"d0962ec0ee700ccb488e9c46139a137c75e30b5f0d0fd18add924e376c837873"} Oct 30 21:39:07.945559 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:07.945470 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09ae27cf-jqpxr" event={"ID":"1597353f-0050-4a88-89df-ccec34a0e2b0","Type":"ContainerStarted","Data":"978f20250008478f213ac31ad02d2c2ec22a4e96cf82d72b59ae629cb7fbd765"} Oct 30 21:39:08.063087 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:08.062995 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09ae27cf-jqpxr" podStartSLOduration=2.062979799 podStartE2EDuration="2.062979799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:08.062329359 +0000 UTC m=+931.804850313" watchObservedRunningTime="2026-10-30 21:39:08.062979799 +0000 UTC m=+931.805500737" Oct 30 21:39:08.166292 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:08.166253 5748 patch_prober.go:29] interesting pod/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 21:39:08.166459 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:08.166307 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9" podUID="26dc7eb4-1548-433d-b7dd-0368d36a466b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 21:39:08.953014 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:08.952971 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09ae27cf-jqpxr" event={"ID":"1597353f-0050-4a88-89df-ccec34a0e2b0","Type":"ContainerDied","Data":"d0962ec0ee700ccb488e9c46139a137c75e30b5f0d0fd18add924e376c837873"} Oct 30 21:39:08.953168 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:08.952885 5748 generic.go:356] "Generic (PLEG): container finished" podID="1597353f-0050-4a88-89df-ccec34a0e2b0" containerID="d0962ec0ee700ccb488e9c46139a137c75e30b5f0d0fd18add924e376c837873" exitCode=0 Oct 30 21:39:08.954765 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:08.954738 5748 generic.go:356] "Generic (PLEG): container finished" podID="2f2cd81a-d648-4baf-94c7-24ffeadce3d7" containerID="22d225f7c37e0858ef6b82f8ac2a5436dab69ac2a03f336291ef8259e768e2a1" exitCode=0 Oct 30 21:39:08.954884 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:08.954770 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96d4216-9zkrg" event={"ID":"2f2cd81a-d648-4baf-94c7-24ffeadce3d7","Type":"ContainerDied","Data":"22d225f7c37e0858ef6b82f8ac2a5436dab69ac2a03f336291ef8259e768e2a1"} Oct 30 21:39:09.166499 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:09.166469 5748 patch_prober.go:29] interesting pod/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 21:39:09.166859 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:09.166530 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9" podUID="26dc7eb4-1548-433d-b7dd-0368d36a466b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 21:39:10.166225 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:10.166194 5748 patch_prober.go:29] interesting pod/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 21:39:10.166366 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:10.166249 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9" podUID="26dc7eb4-1548-433d-b7dd-0368d36a466b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 21:39:10.192626 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:10.192600 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09ae27cf-jqpxr" Oct 30 21:39:10.195607 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:10.195586 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96d4216-9zkrg" Oct 30 21:39:10.297001 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:10.296883 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1597353f-0050-4a88-89df-ccec34a0e2b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1597353f-0050-4a88-89df-ccec34a0e2b0-maas-api-openshift-ca\") pod \"1597353f-0050-4a88-89df-ccec34a0e2b0\" (UID: \"1597353f-0050-4a88-89df-ccec34a0e2b0\") " Oct 30 21:39:10.297156 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:10.297039 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f2cd81a-d648-4baf-94c7-24ffeadce3d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f2cd81a-d648-4baf-94c7-24ffeadce3d7-maas-api-openshift-ca\") pod \"2f2cd81a-d648-4baf-94c7-24ffeadce3d7\" (UID: \"2f2cd81a-d648-4baf-94c7-24ffeadce3d7\") " Oct 30 21:39:10.297156 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:10.297090 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f2cd81a-d648-4baf-94c7-24ffeadce3d7-kube-api-access-tgz5n\" (UniqueName: \"kubernetes.io/projected/2f2cd81a-d648-4baf-94c7-24ffeadce3d7-kube-api-access-tgz5n\") pod \"2f2cd81a-d648-4baf-94c7-24ffeadce3d7\" (UID: \"2f2cd81a-d648-4baf-94c7-24ffeadce3d7\") " Oct 30 21:39:10.297156 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:10.297151 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1597353f-0050-4a88-89df-ccec34a0e2b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1597353f-0050-4a88-89df-ccec34a0e2b0-maas-api-xks-ca\") pod \"1597353f-0050-4a88-89df-ccec34a0e2b0\" (UID: \"1597353f-0050-4a88-89df-ccec34a0e2b0\") " Oct 30 21:39:10.297337 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:10.297173 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1597353f-0050-4a88-89df-ccec34a0e2b0-kube-api-access-9qmkj\" (UniqueName: \"kubernetes.io/projected/1597353f-0050-4a88-89df-ccec34a0e2b0-kube-api-access-9qmkj\") pod \"1597353f-0050-4a88-89df-ccec34a0e2b0\" (UID: \"1597353f-0050-4a88-89df-ccec34a0e2b0\") " Oct 30 21:39:10.297337 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:10.297221 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f2cd81a-d648-4baf-94c7-24ffeadce3d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f2cd81a-d648-4baf-94c7-24ffeadce3d7-maas-api-xks-ca\") pod \"2f2cd81a-d648-4baf-94c7-24ffeadce3d7\" (UID: \"2f2cd81a-d648-4baf-94c7-24ffeadce3d7\") " Oct 30 21:39:10.297337 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:10.297248 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1597353f-0050-4a88-89df-ccec34a0e2b0-maas-api-openshift-ca" pod "1597353f-0050-4a88-89df-ccec34a0e2b0" (UID: "1597353f-0050-4a88-89df-ccec34a0e2b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:39:10.297507 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:10.297481 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f2cd81a-d648-4baf-94c7-24ffeadce3d7-maas-api-openshift-ca" pod "2f2cd81a-d648-4baf-94c7-24ffeadce3d7" (UID: "2f2cd81a-d648-4baf-94c7-24ffeadce3d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:39:10.297648 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:10.297622 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1597353f-0050-4a88-89df-ccec34a0e2b0-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:10.297773 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:10.297655 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f2cd81a-d648-4baf-94c7-24ffeadce3d7-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:10.299493 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:10.299448 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f2cd81a-d648-4baf-94c7-24ffeadce3d7-kube-api-access-tgz5n" pod "2f2cd81a-d648-4baf-94c7-24ffeadce3d7" (UID: "2f2cd81a-d648-4baf-94c7-24ffeadce3d7"). InnerVolumeSpecName "kube-api-access-tgz5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:39:10.299613 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:10.299523 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f2cd81a-d648-4baf-94c7-24ffeadce3d7-maas-api-xks-ca" pod "2f2cd81a-d648-4baf-94c7-24ffeadce3d7" (UID: "2f2cd81a-d648-4baf-94c7-24ffeadce3d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:39:10.310921 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:10.310238 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1597353f-0050-4a88-89df-ccec34a0e2b0-kube-api-access-9qmkj" pod "1597353f-0050-4a88-89df-ccec34a0e2b0" (UID: "1597353f-0050-4a88-89df-ccec34a0e2b0"). InnerVolumeSpecName "kube-api-access-9qmkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:39:10.334929 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:10.332107 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1597353f-0050-4a88-89df-ccec34a0e2b0-maas-api-xks-ca" pod "1597353f-0050-4a88-89df-ccec34a0e2b0" (UID: "1597353f-0050-4a88-89df-ccec34a0e2b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:39:10.398404 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:10.398364 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgz5n\" (UniqueName: \"kubernetes.io/projected/2f2cd81a-d648-4baf-94c7-24ffeadce3d7-kube-api-access-tgz5n\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:10.398404 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:10.398400 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1597353f-0050-4a88-89df-ccec34a0e2b0-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:10.398404 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:10.398411 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qmkj\" (UniqueName: \"kubernetes.io/projected/1597353f-0050-4a88-89df-ccec34a0e2b0-kube-api-access-9qmkj\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:10.398642 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:10.398420 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f2cd81a-d648-4baf-94c7-24ffeadce3d7-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:10.975782 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:10.975751 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09ae27cf-jqpxr" Oct 30 21:39:10.975962 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:10.975775 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09ae27cf-jqpxr" event={"ID":"1597353f-0050-4a88-89df-ccec34a0e2b0","Type":"ContainerDied","Data":"978f20250008478f213ac31ad02d2c2ec22a4e96cf82d72b59ae629cb7fbd765"} Oct 30 21:39:10.975962 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:10.975829 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="978f20250008478f213ac31ad02d2c2ec22a4e96cf82d72b59ae629cb7fbd765" Oct 30 21:39:10.977437 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:10.977414 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96d4216-9zkrg" Oct 30 21:39:10.977590 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:10.977416 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96d4216-9zkrg" event={"ID":"2f2cd81a-d648-4baf-94c7-24ffeadce3d7","Type":"ContainerDied","Data":"236eeeaa6dceb1e829908cabf552d55499a203966b5cadf814b7d3e112f68b84"} Oct 30 21:39:10.977590 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:10.977554 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="236eeeaa6dceb1e829908cabf552d55499a203966b5cadf814b7d3e112f68b84" Oct 30 21:39:11.384607 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:11.384413 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9" Oct 30 21:39:11.385121 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:11.385020 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9" Oct 30 21:39:11.386329 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:11.386307 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9" Oct 30 21:39:14.443925 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:14.442612 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c56d9ab2-j4wwt"] Oct 30 21:39:14.443925 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:14.443734 5748 cpu_manager.go:414] "containerMap: removing container" podUID="2f2cd81a-d648-4baf-94c7-24ffeadce3d7" containerName="revoke-keys" Oct 30 21:39:14.443925 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:14.443758 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f2cd81a-d648-4baf-94c7-24ffeadce3d7" containerName="revoke-keys" Oct 30 21:39:14.443925 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:14.443807 5748 cpu_manager.go:414] "containerMap: removing container" podUID="a47cdb8d-3422-4853-b26e-6aa87fedfb4b" containerName="revoke-keys" Oct 30 21:39:14.443925 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:14.443816 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a47cdb8d-3422-4853-b26e-6aa87fedfb4b" containerName="revoke-keys" Oct 30 21:39:14.443925 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:14.443839 5748 cpu_manager.go:414] "containerMap: removing container" podUID="1597353f-0050-4a88-89df-ccec34a0e2b0" containerName="revoke-keys" Oct 30 21:39:14.443925 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:14.443847 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1597353f-0050-4a88-89df-ccec34a0e2b0" containerName="revoke-keys" Oct 30 21:39:14.444681 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:14.444099 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f2cd81a-d648-4baf-94c7-24ffeadce3d7" containerName="revoke-keys" Oct 30 21:39:14.444681 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:14.444119 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="a47cdb8d-3422-4853-b26e-6aa87fedfb4b" containerName="revoke-keys" Oct 30 21:39:14.444681 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:14.444135 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="1597353f-0050-4a88-89df-ccec34a0e2b0" containerName="revoke-keys" Oct 30 21:39:14.449769 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:14.449740 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c56d9ab2-j4wwt" Oct 30 21:39:14.503315 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:14.503280 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c56d9ab2-j4wwt"] Oct 30 21:39:14.549743 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:14.549706 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:39:14.549743 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:14.549707 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:39:14.549743 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:14.549716 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:39:14.550408 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:14.550389 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 21:39:14.554962 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:14.554935 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqtl6\" (UniqueName: \"kubernetes.io/projected/80c3b0cc-5eb0-4b50-88f3-6e37bbb7297e-kube-api-access-zqtl6\") pod \"maas-api-revoke-sub-c56d9ab2-j4wwt\" (UID: \"80c3b0cc-5eb0-4b50-88f3-6e37bbb7297e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c56d9ab2-j4wwt" Oct 30 21:39:14.555108 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:14.554985 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80c3b0cc-5eb0-4b50-88f3-6e37bbb7297e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c56d9ab2-j4wwt\" (UID: \"80c3b0cc-5eb0-4b50-88f3-6e37bbb7297e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c56d9ab2-j4wwt" Oct 30 21:39:14.555108 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:14.555098 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80c3b0cc-5eb0-4b50-88f3-6e37bbb7297e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c56d9ab2-j4wwt\" (UID: \"80c3b0cc-5eb0-4b50-88f3-6e37bbb7297e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c56d9ab2-j4wwt" Oct 30 21:39:14.656352 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:14.656316 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqtl6\" (UniqueName: \"kubernetes.io/projected/80c3b0cc-5eb0-4b50-88f3-6e37bbb7297e-kube-api-access-zqtl6\") pod \"maas-api-revoke-sub-c56d9ab2-j4wwt\" (UID: \"80c3b0cc-5eb0-4b50-88f3-6e37bbb7297e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c56d9ab2-j4wwt" Oct 30 21:39:14.656533 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:14.656387 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80c3b0cc-5eb0-4b50-88f3-6e37bbb7297e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c56d9ab2-j4wwt\" (UID: \"80c3b0cc-5eb0-4b50-88f3-6e37bbb7297e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c56d9ab2-j4wwt" Oct 30 21:39:14.656533 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:14.656443 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80c3b0cc-5eb0-4b50-88f3-6e37bbb7297e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c56d9ab2-j4wwt\" (UID: \"80c3b0cc-5eb0-4b50-88f3-6e37bbb7297e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c56d9ab2-j4wwt" Oct 30 21:39:14.657124 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:14.657099 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80c3b0cc-5eb0-4b50-88f3-6e37bbb7297e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c56d9ab2-j4wwt\" (UID: \"80c3b0cc-5eb0-4b50-88f3-6e37bbb7297e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c56d9ab2-j4wwt" Oct 30 21:39:14.658865 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:14.658835 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80c3b0cc-5eb0-4b50-88f3-6e37bbb7297e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c56d9ab2-j4wwt\" (UID: \"80c3b0cc-5eb0-4b50-88f3-6e37bbb7297e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c56d9ab2-j4wwt" Oct 30 21:39:14.717812 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:14.717724 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqtl6\" (UniqueName: \"kubernetes.io/projected/80c3b0cc-5eb0-4b50-88f3-6e37bbb7297e-kube-api-access-zqtl6\") pod \"maas-api-revoke-sub-c56d9ab2-j4wwt\" (UID: \"80c3b0cc-5eb0-4b50-88f3-6e37bbb7297e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c56d9ab2-j4wwt" Oct 30 21:39:14.763707 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:14.763672 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c56d9ab2-j4wwt" Oct 30 21:39:15.140967 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:15.140938 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c56d9ab2-j4wwt"] Oct 30 21:39:15.142880 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:39:15.142852 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80c3b0cc_5eb0_4b50_88f3_6e37bbb7297e.slice/crio-ce5c0b8fd0c8356c01d4c64da6f35eb9e6fcbb7f40cd4a526834670ede6faaa7 WatchSource:0}: Error finding container ce5c0b8fd0c8356c01d4c64da6f35eb9e6fcbb7f40cd4a526834670ede6faaa7: Status 404 returned error can't find the container with id ce5c0b8fd0c8356c01d4c64da6f35eb9e6fcbb7f40cd4a526834670ede6faaa7 Oct 30 21:39:16.010629 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:16.010595 5748 generic.go:356] "Generic (PLEG): container finished" podID="80c3b0cc-5eb0-4b50-88f3-6e37bbb7297e" containerID="4815dce7efedf569d0f7a2be749484e87c76202a0f35b07c489eddefc4a18a33" exitCode=0 Oct 30 21:39:16.011046 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:16.010676 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c56d9ab2-j4wwt" event={"ID":"80c3b0cc-5eb0-4b50-88f3-6e37bbb7297e","Type":"ContainerDied","Data":"4815dce7efedf569d0f7a2be749484e87c76202a0f35b07c489eddefc4a18a33"} Oct 30 21:39:16.011046 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:16.010745 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c56d9ab2-j4wwt" event={"ID":"80c3b0cc-5eb0-4b50-88f3-6e37bbb7297e","Type":"ContainerStarted","Data":"ce5c0b8fd0c8356c01d4c64da6f35eb9e6fcbb7f40cd4a526834670ede6faaa7"} Oct 30 21:39:17.160359 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:17.160328 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c56d9ab2-j4wwt" Oct 30 21:39:17.286942 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:17.286845 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80c3b0cc-5eb0-4b50-88f3-6e37bbb7297e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80c3b0cc-5eb0-4b50-88f3-6e37bbb7297e-maas-api-openshift-ca\") pod \"80c3b0cc-5eb0-4b50-88f3-6e37bbb7297e\" (UID: \"80c3b0cc-5eb0-4b50-88f3-6e37bbb7297e\") " Oct 30 21:39:17.287130 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:17.286961 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80c3b0cc-5eb0-4b50-88f3-6e37bbb7297e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80c3b0cc-5eb0-4b50-88f3-6e37bbb7297e-maas-api-xks-ca\") pod \"80c3b0cc-5eb0-4b50-88f3-6e37bbb7297e\" (UID: \"80c3b0cc-5eb0-4b50-88f3-6e37bbb7297e\") " Oct 30 21:39:17.287130 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:17.287000 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80c3b0cc-5eb0-4b50-88f3-6e37bbb7297e-kube-api-access-zqtl6\" (UniqueName: \"kubernetes.io/projected/80c3b0cc-5eb0-4b50-88f3-6e37bbb7297e-kube-api-access-zqtl6\") pod \"80c3b0cc-5eb0-4b50-88f3-6e37bbb7297e\" (UID: \"80c3b0cc-5eb0-4b50-88f3-6e37bbb7297e\") " Oct 30 21:39:17.287330 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:17.287290 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80c3b0cc-5eb0-4b50-88f3-6e37bbb7297e-maas-api-openshift-ca" pod "80c3b0cc-5eb0-4b50-88f3-6e37bbb7297e" (UID: "80c3b0cc-5eb0-4b50-88f3-6e37bbb7297e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:39:17.287481 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:17.287460 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80c3b0cc-5eb0-4b50-88f3-6e37bbb7297e-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:17.289343 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:17.289312 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80c3b0cc-5eb0-4b50-88f3-6e37bbb7297e-kube-api-access-zqtl6" pod "80c3b0cc-5eb0-4b50-88f3-6e37bbb7297e" (UID: "80c3b0cc-5eb0-4b50-88f3-6e37bbb7297e"). InnerVolumeSpecName "kube-api-access-zqtl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:39:17.289535 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:17.289505 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80c3b0cc-5eb0-4b50-88f3-6e37bbb7297e-maas-api-xks-ca" pod "80c3b0cc-5eb0-4b50-88f3-6e37bbb7297e" (UID: "80c3b0cc-5eb0-4b50-88f3-6e37bbb7297e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:39:17.388839 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:17.388804 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80c3b0cc-5eb0-4b50-88f3-6e37bbb7297e-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:17.388839 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:17.388833 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqtl6\" (UniqueName: \"kubernetes.io/projected/80c3b0cc-5eb0-4b50-88f3-6e37bbb7297e-kube-api-access-zqtl6\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:18.023481 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:18.023441 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c56d9ab2-j4wwt" event={"ID":"80c3b0cc-5eb0-4b50-88f3-6e37bbb7297e","Type":"ContainerDied","Data":"ce5c0b8fd0c8356c01d4c64da6f35eb9e6fcbb7f40cd4a526834670ede6faaa7"} Oct 30 21:39:18.023481 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:18.023471 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c56d9ab2-j4wwt" Oct 30 21:39:18.023695 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:18.023492 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce5c0b8fd0c8356c01d4c64da6f35eb9e6fcbb7f40cd4a526834670ede6faaa7" Oct 30 21:39:18.995617 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:18.995576 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3399cba9-7rmq7"] Oct 30 21:39:18.996104 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:18.996089 5748 cpu_manager.go:414] "containerMap: removing container" podUID="80c3b0cc-5eb0-4b50-88f3-6e37bbb7297e" containerName="revoke-keys" Oct 30 21:39:18.996165 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:18.996106 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80c3b0cc-5eb0-4b50-88f3-6e37bbb7297e" containerName="revoke-keys" Oct 30 21:39:18.996214 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:18.996205 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="80c3b0cc-5eb0-4b50-88f3-6e37bbb7297e" containerName="revoke-keys" Oct 30 21:39:18.999586 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:18.999569 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3399cba9-7rmq7" Oct 30 21:39:19.055066 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:19.054843 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:39:19.105154 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:19.105117 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:39:19.105346 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:19.105229 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:39:19.105393 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:19.105340 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 21:39:19.107611 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:19.107591 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3399cba9-7rmq7"] Oct 30 21:39:19.107883 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:19.107861 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/384173e5-7d05-4873-915d-a333af9f35e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3399cba9-7rmq7\" (UID: \"384173e5-7d05-4873-915d-a333af9f35e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3399cba9-7rmq7" Oct 30 21:39:19.107987 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:19.107974 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/384173e5-7d05-4873-915d-a333af9f35e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3399cba9-7rmq7\" (UID: \"384173e5-7d05-4873-915d-a333af9f35e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3399cba9-7rmq7" Oct 30 21:39:19.108025 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:19.107999 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2xvc\" (UniqueName: \"kubernetes.io/projected/384173e5-7d05-4873-915d-a333af9f35e0-kube-api-access-c2xvc\") pod \"maas-api-revoke-sub-3399cba9-7rmq7\" (UID: \"384173e5-7d05-4873-915d-a333af9f35e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3399cba9-7rmq7" Oct 30 21:39:19.209087 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:19.209050 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/384173e5-7d05-4873-915d-a333af9f35e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3399cba9-7rmq7\" (UID: \"384173e5-7d05-4873-915d-a333af9f35e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3399cba9-7rmq7" Oct 30 21:39:19.209246 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:19.209118 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/384173e5-7d05-4873-915d-a333af9f35e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3399cba9-7rmq7\" (UID: \"384173e5-7d05-4873-915d-a333af9f35e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3399cba9-7rmq7" Oct 30 21:39:19.209336 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:19.209276 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2xvc\" (UniqueName: \"kubernetes.io/projected/384173e5-7d05-4873-915d-a333af9f35e0-kube-api-access-c2xvc\") pod \"maas-api-revoke-sub-3399cba9-7rmq7\" (UID: \"384173e5-7d05-4873-915d-a333af9f35e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3399cba9-7rmq7" Oct 30 21:39:19.209959 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:19.209938 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/384173e5-7d05-4873-915d-a333af9f35e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3399cba9-7rmq7\" (UID: \"384173e5-7d05-4873-915d-a333af9f35e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3399cba9-7rmq7" Oct 30 21:39:19.211747 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:19.211725 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/384173e5-7d05-4873-915d-a333af9f35e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3399cba9-7rmq7\" (UID: \"384173e5-7d05-4873-915d-a333af9f35e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3399cba9-7rmq7" Oct 30 21:39:19.271465 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:19.271401 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2xvc\" (UniqueName: \"kubernetes.io/projected/384173e5-7d05-4873-915d-a333af9f35e0-kube-api-access-c2xvc\") pod \"maas-api-revoke-sub-3399cba9-7rmq7\" (UID: \"384173e5-7d05-4873-915d-a333af9f35e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3399cba9-7rmq7" Oct 30 21:39:19.317811 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:19.317778 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3399cba9-7rmq7" Oct 30 21:39:19.699429 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:19.699265 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3399cba9-7rmq7"] Oct 30 21:39:19.702456 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:39:19.702404 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod384173e5_7d05_4873_915d_a333af9f35e0.slice/crio-1a93971905e892f72a0bc3af4a008cd7cc460313a680badd390fbf6079ad8512 WatchSource:0}: Error finding container 1a93971905e892f72a0bc3af4a008cd7cc460313a680badd390fbf6079ad8512: Status 404 returned error can't find the container with id 1a93971905e892f72a0bc3af4a008cd7cc460313a680badd390fbf6079ad8512 Oct 30 21:39:20.037465 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:20.037430 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3399cba9-7rmq7" event={"ID":"384173e5-7d05-4873-915d-a333af9f35e0","Type":"ContainerStarted","Data":"1093f1c9cb68bf0c843b2f2a223ae0dc36f925393964820ad82bceb5849849ad"} Oct 30 21:39:20.037923 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:20.037488 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3399cba9-7rmq7" event={"ID":"384173e5-7d05-4873-915d-a333af9f35e0","Type":"ContainerStarted","Data":"1a93971905e892f72a0bc3af4a008cd7cc460313a680badd390fbf6079ad8512"} Oct 30 21:39:20.156032 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:20.155978 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3399cba9-7rmq7" podStartSLOduration=2.15596464 podStartE2EDuration="2.15596464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:20.154125847 +0000 UTC m=+943.896646783" watchObservedRunningTime="2026-10-30 21:39:20.15596464 +0000 UTC m=+943.898485574" Oct 30 21:39:20.656186 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:20.656145 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd"] Oct 30 21:39:20.660390 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:20.660367 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd" Oct 30 21:39:20.713644 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:20.713617 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-691598-openshift-default-dockercfg-trswq\"" Oct 30 21:39:20.765835 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:20.765732 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd"] Oct 30 21:39:20.826004 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:20.825965 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/890f0ec3-96a5-4ea5-8137-e7166101e49b-istio-envoy\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-87njd\" (UID: \"890f0ec3-96a5-4ea5-8137-e7166101e49b\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd" Oct 30 21:39:20.826004 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:20.826007 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vxqv\" (UniqueName: \"kubernetes.io/projected/890f0ec3-96a5-4ea5-8137-e7166101e49b-kube-api-access-6vxqv\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-87njd\" (UID: \"890f0ec3-96a5-4ea5-8137-e7166101e49b\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd" Oct 30 21:39:20.826216 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:20.826127 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/890f0ec3-96a5-4ea5-8137-e7166101e49b-istio-token\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-87njd\" (UID: \"890f0ec3-96a5-4ea5-8137-e7166101e49b\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd" Oct 30 21:39:20.826216 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:20.826175 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/890f0ec3-96a5-4ea5-8137-e7166101e49b-workload-socket\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-87njd\" (UID: \"890f0ec3-96a5-4ea5-8137-e7166101e49b\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd" Oct 30 21:39:20.826216 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:20.826200 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/890f0ec3-96a5-4ea5-8137-e7166101e49b-istiod-ca-cert\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-87njd\" (UID: \"890f0ec3-96a5-4ea5-8137-e7166101e49b\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd" Oct 30 21:39:20.826325 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:20.826237 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/890f0ec3-96a5-4ea5-8137-e7166101e49b-istio-podinfo\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-87njd\" (UID: \"890f0ec3-96a5-4ea5-8137-e7166101e49b\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd" Oct 30 21:39:20.826325 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:20.826315 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/890f0ec3-96a5-4ea5-8137-e7166101e49b-workload-certs\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-87njd\" (UID: \"890f0ec3-96a5-4ea5-8137-e7166101e49b\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd" Oct 30 21:39:20.826393 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:20.826345 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/890f0ec3-96a5-4ea5-8137-e7166101e49b-credential-socket\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-87njd\" (UID: \"890f0ec3-96a5-4ea5-8137-e7166101e49b\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd" Oct 30 21:39:20.826393 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:20.826366 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/890f0ec3-96a5-4ea5-8137-e7166101e49b-istio-data\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-87njd\" (UID: \"890f0ec3-96a5-4ea5-8137-e7166101e49b\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd" Oct 30 21:39:20.927810 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:20.927768 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/890f0ec3-96a5-4ea5-8137-e7166101e49b-istio-envoy\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-87njd\" (UID: \"890f0ec3-96a5-4ea5-8137-e7166101e49b\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd" Oct 30 21:39:20.927810 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:20.927812 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vxqv\" (UniqueName: \"kubernetes.io/projected/890f0ec3-96a5-4ea5-8137-e7166101e49b-kube-api-access-6vxqv\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-87njd\" (UID: \"890f0ec3-96a5-4ea5-8137-e7166101e49b\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd" Oct 30 21:39:20.928095 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:20.927850 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/890f0ec3-96a5-4ea5-8137-e7166101e49b-istio-token\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-87njd\" (UID: \"890f0ec3-96a5-4ea5-8137-e7166101e49b\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd" Oct 30 21:39:20.928095 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:20.927878 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/890f0ec3-96a5-4ea5-8137-e7166101e49b-workload-socket\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-87njd\" (UID: \"890f0ec3-96a5-4ea5-8137-e7166101e49b\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd" Oct 30 21:39:20.928095 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:20.927928 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/890f0ec3-96a5-4ea5-8137-e7166101e49b-istiod-ca-cert\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-87njd\" (UID: \"890f0ec3-96a5-4ea5-8137-e7166101e49b\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd" Oct 30 21:39:20.928095 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:20.927970 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/890f0ec3-96a5-4ea5-8137-e7166101e49b-istio-podinfo\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-87njd\" (UID: \"890f0ec3-96a5-4ea5-8137-e7166101e49b\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd" Oct 30 21:39:20.928095 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:20.928021 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/890f0ec3-96a5-4ea5-8137-e7166101e49b-workload-certs\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-87njd\" (UID: \"890f0ec3-96a5-4ea5-8137-e7166101e49b\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd" Oct 30 21:39:20.928095 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:20.928062 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/890f0ec3-96a5-4ea5-8137-e7166101e49b-credential-socket\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-87njd\" (UID: \"890f0ec3-96a5-4ea5-8137-e7166101e49b\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd" Oct 30 21:39:20.928095 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:20.928088 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/890f0ec3-96a5-4ea5-8137-e7166101e49b-istio-data\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-87njd\" (UID: \"890f0ec3-96a5-4ea5-8137-e7166101e49b\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd" Oct 30 21:39:20.928497 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:20.928443 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/890f0ec3-96a5-4ea5-8137-e7166101e49b-workload-socket\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-87njd\" (UID: \"890f0ec3-96a5-4ea5-8137-e7166101e49b\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd" Oct 30 21:39:20.928600 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:20.928506 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/890f0ec3-96a5-4ea5-8137-e7166101e49b-istio-data\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-87njd\" (UID: \"890f0ec3-96a5-4ea5-8137-e7166101e49b\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd" Oct 30 21:39:20.928600 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:20.928573 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/890f0ec3-96a5-4ea5-8137-e7166101e49b-credential-socket\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-87njd\" (UID: \"890f0ec3-96a5-4ea5-8137-e7166101e49b\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd" Oct 30 21:39:20.928689 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:20.928607 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/890f0ec3-96a5-4ea5-8137-e7166101e49b-workload-certs\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-87njd\" (UID: \"890f0ec3-96a5-4ea5-8137-e7166101e49b\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd" Oct 30 21:39:20.928750 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:20.928730 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/890f0ec3-96a5-4ea5-8137-e7166101e49b-istiod-ca-cert\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-87njd\" (UID: \"890f0ec3-96a5-4ea5-8137-e7166101e49b\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd" Oct 30 21:39:20.930223 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:20.930206 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/890f0ec3-96a5-4ea5-8137-e7166101e49b-istio-envoy\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-87njd\" (UID: \"890f0ec3-96a5-4ea5-8137-e7166101e49b\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd" Oct 30 21:39:20.930555 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:20.930532 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/890f0ec3-96a5-4ea5-8137-e7166101e49b-istio-podinfo\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-87njd\" (UID: \"890f0ec3-96a5-4ea5-8137-e7166101e49b\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd" Oct 30 21:39:20.990420 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:20.990387 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/890f0ec3-96a5-4ea5-8137-e7166101e49b-istio-token\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-87njd\" (UID: \"890f0ec3-96a5-4ea5-8137-e7166101e49b\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd" Oct 30 21:39:20.990633 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:20.990608 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vxqv\" (UniqueName: \"kubernetes.io/projected/890f0ec3-96a5-4ea5-8137-e7166101e49b-kube-api-access-6vxqv\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-87njd\" (UID: \"890f0ec3-96a5-4ea5-8137-e7166101e49b\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd" Oct 30 21:39:21.044782 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:21.044693 5748 generic.go:356] "Generic (PLEG): container finished" podID="384173e5-7d05-4873-915d-a333af9f35e0" containerID="1093f1c9cb68bf0c843b2f2a223ae0dc36f925393964820ad82bceb5849849ad" exitCode=0 Oct 30 21:39:21.045232 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:21.044774 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3399cba9-7rmq7" event={"ID":"384173e5-7d05-4873-915d-a333af9f35e0","Type":"ContainerDied","Data":"1093f1c9cb68bf0c843b2f2a223ae0dc36f925393964820ad82bceb5849849ad"} Oct 30 21:39:21.274099 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:21.274060 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd" Oct 30 21:39:21.705274 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:21.705249 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd"] Oct 30 21:39:21.707235 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:39:21.707205 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod890f0ec3_96a5_4ea5_8137_e7166101e49b.slice/crio-08ceb122d36cf9a30130285c301cde061f7d0aa0270882bab03f263b7236385b WatchSource:0}: Error finding container 08ceb122d36cf9a30130285c301cde061f7d0aa0270882bab03f263b7236385b: Status 404 returned error can't find the container with id 08ceb122d36cf9a30130285c301cde061f7d0aa0270882bab03f263b7236385b Oct 30 21:39:21.709644 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:21.709611 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:39:21.709747 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:21.709689 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:39:21.709747 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:21.709734 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:39:22.054984 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:22.054180 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd" event={"ID":"890f0ec3-96a5-4ea5-8137-e7166101e49b","Type":"ContainerStarted","Data":"2c57766216776521d66068bcbee99f6c0c443ad2eec51a3b2013b91d99e201d0"} Oct 30 21:39:22.054984 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:22.054312 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd" event={"ID":"890f0ec3-96a5-4ea5-8137-e7166101e49b","Type":"ContainerStarted","Data":"08ceb122d36cf9a30130285c301cde061f7d0aa0270882bab03f263b7236385b"} Oct 30 21:39:22.177013 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:22.176958 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd" podStartSLOduration=2.176944173 podStartE2EDuration="2.176944173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:22.175270572 +0000 UTC m=+945.917791511" watchObservedRunningTime="2026-10-30 21:39:22.176944173 +0000 UTC m=+945.919465107" Oct 30 21:39:22.191805 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:22.191775 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3399cba9-7rmq7" Oct 30 21:39:22.274968 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:22.274931 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd" Oct 30 21:39:22.276644 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:22.276614 5748 patch_prober.go:29] interesting pod/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 21:39:22.276769 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:22.276666 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd" podUID="890f0ec3-96a5-4ea5-8137-e7166101e49b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 21:39:22.343416 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:22.343328 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/384173e5-7d05-4873-915d-a333af9f35e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/384173e5-7d05-4873-915d-a333af9f35e0-maas-api-openshift-ca\") pod \"384173e5-7d05-4873-915d-a333af9f35e0\" (UID: \"384173e5-7d05-4873-915d-a333af9f35e0\") " Oct 30 21:39:22.343579 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:22.343487 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/384173e5-7d05-4873-915d-a333af9f35e0-kube-api-access-c2xvc\" (UniqueName: \"kubernetes.io/projected/384173e5-7d05-4873-915d-a333af9f35e0-kube-api-access-c2xvc\") pod \"384173e5-7d05-4873-915d-a333af9f35e0\" (UID: \"384173e5-7d05-4873-915d-a333af9f35e0\") " Oct 30 21:39:22.343647 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:22.343597 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/384173e5-7d05-4873-915d-a333af9f35e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/384173e5-7d05-4873-915d-a333af9f35e0-maas-api-xks-ca\") pod \"384173e5-7d05-4873-915d-a333af9f35e0\" (UID: \"384173e5-7d05-4873-915d-a333af9f35e0\") " Oct 30 21:39:22.343741 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:22.343719 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/384173e5-7d05-4873-915d-a333af9f35e0-maas-api-openshift-ca" pod "384173e5-7d05-4873-915d-a333af9f35e0" (UID: "384173e5-7d05-4873-915d-a333af9f35e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:39:22.343955 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:22.343939 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/384173e5-7d05-4873-915d-a333af9f35e0-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:22.345689 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:22.345656 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/384173e5-7d05-4873-915d-a333af9f35e0-maas-api-xks-ca" pod "384173e5-7d05-4873-915d-a333af9f35e0" (UID: "384173e5-7d05-4873-915d-a333af9f35e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:39:22.345807 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:22.345785 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/384173e5-7d05-4873-915d-a333af9f35e0-kube-api-access-c2xvc" pod "384173e5-7d05-4873-915d-a333af9f35e0" (UID: "384173e5-7d05-4873-915d-a333af9f35e0"). InnerVolumeSpecName "kube-api-access-c2xvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:39:22.445446 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:22.445417 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/384173e5-7d05-4873-915d-a333af9f35e0-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:22.445446 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:22.445446 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2xvc\" (UniqueName: \"kubernetes.io/projected/384173e5-7d05-4873-915d-a333af9f35e0-kube-api-access-c2xvc\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:23.061003 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:23.060968 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3399cba9-7rmq7" Oct 30 21:39:23.061471 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:23.061012 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3399cba9-7rmq7" event={"ID":"384173e5-7d05-4873-915d-a333af9f35e0","Type":"ContainerDied","Data":"1a93971905e892f72a0bc3af4a008cd7cc460313a680badd390fbf6079ad8512"} Oct 30 21:39:23.061471 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:23.061064 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a93971905e892f72a0bc3af4a008cd7cc460313a680badd390fbf6079ad8512" Oct 30 21:39:23.275190 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:23.275155 5748 patch_prober.go:29] interesting pod/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 21:39:23.275368 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:23.275220 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd" podUID="890f0ec3-96a5-4ea5-8137-e7166101e49b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 21:39:24.274697 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:24.274662 5748 patch_prober.go:29] interesting pod/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 21:39:24.275125 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:24.274752 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd" podUID="890f0ec3-96a5-4ea5-8137-e7166101e49b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 21:39:24.470253 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:24.470215 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b425e42-2fnd9"] Oct 30 21:39:24.470807 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:24.470789 5748 cpu_manager.go:414] "containerMap: removing container" podUID="384173e5-7d05-4873-915d-a333af9f35e0" containerName="revoke-keys" Oct 30 21:39:24.470807 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:24.470809 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="384173e5-7d05-4873-915d-a333af9f35e0" containerName="revoke-keys" Oct 30 21:39:24.470995 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:24.470951 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="384173e5-7d05-4873-915d-a333af9f35e0" containerName="revoke-keys" Oct 30 21:39:24.474576 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:24.474551 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b425e42-2fnd9" Oct 30 21:39:24.580234 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:24.580134 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:39:24.580234 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:24.580186 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:39:24.580234 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:24.580186 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:39:24.580234 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:24.580219 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 21:39:24.582822 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:24.582795 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b425e42-2fnd9"] Oct 30 21:39:24.635710 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:24.635674 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd95a157-qmjvl"] Oct 30 21:39:24.639345 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:24.639327 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd95a157-qmjvl" Oct 30 21:39:24.660205 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:24.660177 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd95a157-qmjvl"] Oct 30 21:39:24.668464 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:24.668435 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdm2s\" (UniqueName: \"kubernetes.io/projected/6b6e60cc-ea7d-4c1e-bef8-3178637efa0b-kube-api-access-pdm2s\") pod \"maas-api-revoke-sub-6b425e42-2fnd9\" (UID: \"6b6e60cc-ea7d-4c1e-bef8-3178637efa0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b425e42-2fnd9" Oct 30 21:39:24.668591 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:24.668515 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b6e60cc-ea7d-4c1e-bef8-3178637efa0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b425e42-2fnd9\" (UID: \"6b6e60cc-ea7d-4c1e-bef8-3178637efa0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b425e42-2fnd9" Oct 30 21:39:24.668644 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:24.668580 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b6e60cc-ea7d-4c1e-bef8-3178637efa0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b425e42-2fnd9\" (UID: \"6b6e60cc-ea7d-4c1e-bef8-3178637efa0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b425e42-2fnd9" Oct 30 21:39:24.769438 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:24.769400 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdm2s\" (UniqueName: \"kubernetes.io/projected/6b6e60cc-ea7d-4c1e-bef8-3178637efa0b-kube-api-access-pdm2s\") pod \"maas-api-revoke-sub-6b425e42-2fnd9\" (UID: \"6b6e60cc-ea7d-4c1e-bef8-3178637efa0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b425e42-2fnd9" Oct 30 21:39:24.769438 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:24.769441 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1acf0f40-6f96-4914-8350-c5e19015968c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd95a157-qmjvl\" (UID: \"1acf0f40-6f96-4914-8350-c5e19015968c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd95a157-qmjvl" Oct 30 21:39:24.769677 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:24.769460 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1acf0f40-6f96-4914-8350-c5e19015968c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd95a157-qmjvl\" (UID: \"1acf0f40-6f96-4914-8350-c5e19015968c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd95a157-qmjvl" Oct 30 21:39:24.769677 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:24.769613 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b6e60cc-ea7d-4c1e-bef8-3178637efa0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b425e42-2fnd9\" (UID: \"6b6e60cc-ea7d-4c1e-bef8-3178637efa0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b425e42-2fnd9" Oct 30 21:39:24.769677 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:24.769644 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b6e60cc-ea7d-4c1e-bef8-3178637efa0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b425e42-2fnd9\" (UID: \"6b6e60cc-ea7d-4c1e-bef8-3178637efa0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b425e42-2fnd9" Oct 30 21:39:24.769832 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:24.769716 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86cj7\" (UniqueName: \"kubernetes.io/projected/1acf0f40-6f96-4914-8350-c5e19015968c-kube-api-access-86cj7\") pod \"maas-api-revoke-sub-bd95a157-qmjvl\" (UID: \"1acf0f40-6f96-4914-8350-c5e19015968c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd95a157-qmjvl" Oct 30 21:39:24.770316 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:24.770288 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b6e60cc-ea7d-4c1e-bef8-3178637efa0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b425e42-2fnd9\" (UID: \"6b6e60cc-ea7d-4c1e-bef8-3178637efa0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b425e42-2fnd9" Oct 30 21:39:24.771951 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:24.771926 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b6e60cc-ea7d-4c1e-bef8-3178637efa0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b425e42-2fnd9\" (UID: \"6b6e60cc-ea7d-4c1e-bef8-3178637efa0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b425e42-2fnd9" Oct 30 21:39:24.836880 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:24.836797 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdm2s\" (UniqueName: \"kubernetes.io/projected/6b6e60cc-ea7d-4c1e-bef8-3178637efa0b-kube-api-access-pdm2s\") pod \"maas-api-revoke-sub-6b425e42-2fnd9\" (UID: \"6b6e60cc-ea7d-4c1e-bef8-3178637efa0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b425e42-2fnd9" Oct 30 21:39:24.870738 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:24.870700 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1acf0f40-6f96-4914-8350-c5e19015968c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd95a157-qmjvl\" (UID: \"1acf0f40-6f96-4914-8350-c5e19015968c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd95a157-qmjvl" Oct 30 21:39:24.870738 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:24.870739 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1acf0f40-6f96-4914-8350-c5e19015968c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd95a157-qmjvl\" (UID: \"1acf0f40-6f96-4914-8350-c5e19015968c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd95a157-qmjvl" Oct 30 21:39:24.870974 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:24.870933 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86cj7\" (UniqueName: \"kubernetes.io/projected/1acf0f40-6f96-4914-8350-c5e19015968c-kube-api-access-86cj7\") pod \"maas-api-revoke-sub-bd95a157-qmjvl\" (UID: \"1acf0f40-6f96-4914-8350-c5e19015968c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd95a157-qmjvl" Oct 30 21:39:24.871447 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:24.871421 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1acf0f40-6f96-4914-8350-c5e19015968c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd95a157-qmjvl\" (UID: \"1acf0f40-6f96-4914-8350-c5e19015968c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd95a157-qmjvl" Oct 30 21:39:24.873113 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:24.873088 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1acf0f40-6f96-4914-8350-c5e19015968c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd95a157-qmjvl\" (UID: \"1acf0f40-6f96-4914-8350-c5e19015968c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd95a157-qmjvl" Oct 30 21:39:24.932628 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:24.932594 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86cj7\" (UniqueName: \"kubernetes.io/projected/1acf0f40-6f96-4914-8350-c5e19015968c-kube-api-access-86cj7\") pod \"maas-api-revoke-sub-bd95a157-qmjvl\" (UID: \"1acf0f40-6f96-4914-8350-c5e19015968c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd95a157-qmjvl" Oct 30 21:39:24.952587 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:24.952561 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd95a157-qmjvl" Oct 30 21:39:25.090443 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:25.090360 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b425e42-2fnd9" Oct 30 21:39:25.275033 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:25.274999 5748 patch_prober.go:29] interesting pod/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 21:39:25.275391 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:25.275055 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd" podUID="890f0ec3-96a5-4ea5-8137-e7166101e49b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 21:39:25.353774 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:25.353748 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd95a157-qmjvl"] Oct 30 21:39:25.355413 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:39:25.355377 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1acf0f40_6f96_4914_8350_c5e19015968c.slice/crio-4a7aeb085b854f40a3b4d79d93de736b10498449617bf9159bf9851c728f8e67 WatchSource:0}: Error finding container 4a7aeb085b854f40a3b4d79d93de736b10498449617bf9159bf9851c728f8e67: Status 404 returned error can't find the container with id 4a7aeb085b854f40a3b4d79d93de736b10498449617bf9159bf9851c728f8e67 Oct 30 21:39:25.492441 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:25.492413 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b425e42-2fnd9"] Oct 30 21:39:25.494531 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:39:25.494499 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b6e60cc_ea7d_4c1e_bef8_3178637efa0b.slice/crio-dcec7350a46dedf678a0190a1b618c06fa7e9f0d4a654eb5f18a5e125e48d4d2 WatchSource:0}: Error finding container dcec7350a46dedf678a0190a1b618c06fa7e9f0d4a654eb5f18a5e125e48d4d2: Status 404 returned error can't find the container with id dcec7350a46dedf678a0190a1b618c06fa7e9f0d4a654eb5f18a5e125e48d4d2 Oct 30 21:39:26.081720 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:26.081606 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b425e42-2fnd9" event={"ID":"6b6e60cc-ea7d-4c1e-bef8-3178637efa0b","Type":"ContainerStarted","Data":"5c0dd171563e3cdd2b99ad1439106487b1e785aeb9f1a0ce12f3f779ae9c9150"} Oct 30 21:39:26.081720 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:26.081708 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b425e42-2fnd9" event={"ID":"6b6e60cc-ea7d-4c1e-bef8-3178637efa0b","Type":"ContainerStarted","Data":"dcec7350a46dedf678a0190a1b618c06fa7e9f0d4a654eb5f18a5e125e48d4d2"} Oct 30 21:39:26.083209 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:26.083180 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd95a157-qmjvl" event={"ID":"1acf0f40-6f96-4914-8350-c5e19015968c","Type":"ContainerStarted","Data":"e0f5bac11a634bac11fdc6677dde434b3cb92d58f118a10d1113f383c3309110"} Oct 30 21:39:26.083325 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:26.083230 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd95a157-qmjvl" event={"ID":"1acf0f40-6f96-4914-8350-c5e19015968c","Type":"ContainerStarted","Data":"4a7aeb085b854f40a3b4d79d93de736b10498449617bf9159bf9851c728f8e67"} Oct 30 21:39:26.275239 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:26.275202 5748 patch_prober.go:29] interesting pod/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 21:39:26.275639 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:26.275260 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd" podUID="890f0ec3-96a5-4ea5-8137-e7166101e49b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 21:39:26.321922 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:26.318617 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b425e42-2fnd9" podStartSLOduration=2.318595153 podStartE2EDuration="2.318595153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:26.198510654 +0000 UTC m=+949.941031588" watchObservedRunningTime="2026-10-30 21:39:26.318595153 +0000 UTC m=+950.061116091" Oct 30 21:39:26.328654 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:26.328353 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd95a157-qmjvl" podStartSLOduration=2.32833608 podStartE2EDuration="2.32833608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:26.327857507 +0000 UTC m=+950.070378443" watchObservedRunningTime="2026-10-30 21:39:26.32833608 +0000 UTC m=+950.070857016" Oct 30 21:39:27.090345 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:27.090309 5748 generic.go:356] "Generic (PLEG): container finished" podID="6b6e60cc-ea7d-4c1e-bef8-3178637efa0b" containerID="5c0dd171563e3cdd2b99ad1439106487b1e785aeb9f1a0ce12f3f779ae9c9150" exitCode=0 Oct 30 21:39:27.090552 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:27.090400 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b425e42-2fnd9" event={"ID":"6b6e60cc-ea7d-4c1e-bef8-3178637efa0b","Type":"ContainerDied","Data":"5c0dd171563e3cdd2b99ad1439106487b1e785aeb9f1a0ce12f3f779ae9c9150"} Oct 30 21:39:27.092038 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:27.092011 5748 generic.go:356] "Generic (PLEG): container finished" podID="1acf0f40-6f96-4914-8350-c5e19015968c" containerID="e0f5bac11a634bac11fdc6677dde434b3cb92d58f118a10d1113f383c3309110" exitCode=0 Oct 30 21:39:27.092153 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:27.092052 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd95a157-qmjvl" event={"ID":"1acf0f40-6f96-4914-8350-c5e19015968c","Type":"ContainerDied","Data":"e0f5bac11a634bac11fdc6677dde434b3cb92d58f118a10d1113f383c3309110"} Oct 30 21:39:27.275407 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:27.275373 5748 patch_prober.go:29] interesting pod/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 21:39:27.275798 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:27.275426 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd" podUID="890f0ec3-96a5-4ea5-8137-e7166101e49b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 21:39:28.260677 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:28.260645 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b425e42-2fnd9" Oct 30 21:39:28.304720 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:28.304694 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b6e60cc-ea7d-4c1e-bef8-3178637efa0b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b6e60cc-ea7d-4c1e-bef8-3178637efa0b-maas-api-openshift-ca\") pod \"6b6e60cc-ea7d-4c1e-bef8-3178637efa0b\" (UID: \"6b6e60cc-ea7d-4c1e-bef8-3178637efa0b\") " Oct 30 21:39:28.305181 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:28.304847 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b6e60cc-ea7d-4c1e-bef8-3178637efa0b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b6e60cc-ea7d-4c1e-bef8-3178637efa0b-maas-api-xks-ca\") pod \"6b6e60cc-ea7d-4c1e-bef8-3178637efa0b\" (UID: \"6b6e60cc-ea7d-4c1e-bef8-3178637efa0b\") " Oct 30 21:39:28.305181 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:28.304925 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b6e60cc-ea7d-4c1e-bef8-3178637efa0b-kube-api-access-pdm2s\" (UniqueName: \"kubernetes.io/projected/6b6e60cc-ea7d-4c1e-bef8-3178637efa0b-kube-api-access-pdm2s\") pod \"6b6e60cc-ea7d-4c1e-bef8-3178637efa0b\" (UID: \"6b6e60cc-ea7d-4c1e-bef8-3178637efa0b\") " Oct 30 21:39:28.305181 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:28.305104 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b6e60cc-ea7d-4c1e-bef8-3178637efa0b-maas-api-openshift-ca" pod "6b6e60cc-ea7d-4c1e-bef8-3178637efa0b" (UID: "6b6e60cc-ea7d-4c1e-bef8-3178637efa0b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:39:28.305344 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:28.305262 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b6e60cc-ea7d-4c1e-bef8-3178637efa0b-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:28.307447 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:28.307418 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b6e60cc-ea7d-4c1e-bef8-3178637efa0b-maas-api-xks-ca" pod "6b6e60cc-ea7d-4c1e-bef8-3178637efa0b" (UID: "6b6e60cc-ea7d-4c1e-bef8-3178637efa0b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:39:28.307447 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:28.307440 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b6e60cc-ea7d-4c1e-bef8-3178637efa0b-kube-api-access-pdm2s" pod "6b6e60cc-ea7d-4c1e-bef8-3178637efa0b" (UID: "6b6e60cc-ea7d-4c1e-bef8-3178637efa0b"). InnerVolumeSpecName "kube-api-access-pdm2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:39:28.311994 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:28.311969 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd95a157-qmjvl" Oct 30 21:39:28.405760 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:28.405724 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1acf0f40-6f96-4914-8350-c5e19015968c-kube-api-access-86cj7\" (UniqueName: \"kubernetes.io/projected/1acf0f40-6f96-4914-8350-c5e19015968c-kube-api-access-86cj7\") pod \"1acf0f40-6f96-4914-8350-c5e19015968c\" (UID: \"1acf0f40-6f96-4914-8350-c5e19015968c\") " Oct 30 21:39:28.405960 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:28.405837 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1acf0f40-6f96-4914-8350-c5e19015968c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1acf0f40-6f96-4914-8350-c5e19015968c-maas-api-xks-ca\") pod \"1acf0f40-6f96-4914-8350-c5e19015968c\" (UID: \"1acf0f40-6f96-4914-8350-c5e19015968c\") " Oct 30 21:39:28.405960 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:28.405946 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1acf0f40-6f96-4914-8350-c5e19015968c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1acf0f40-6f96-4914-8350-c5e19015968c-maas-api-openshift-ca\") pod \"1acf0f40-6f96-4914-8350-c5e19015968c\" (UID: \"1acf0f40-6f96-4914-8350-c5e19015968c\") " Oct 30 21:39:28.406541 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:28.406339 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b6e60cc-ea7d-4c1e-bef8-3178637efa0b-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:28.406541 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:28.406370 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdm2s\" (UniqueName: \"kubernetes.io/projected/6b6e60cc-ea7d-4c1e-bef8-3178637efa0b-kube-api-access-pdm2s\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:28.406541 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:28.406475 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1acf0f40-6f96-4914-8350-c5e19015968c-maas-api-openshift-ca" pod "1acf0f40-6f96-4914-8350-c5e19015968c" (UID: "1acf0f40-6f96-4914-8350-c5e19015968c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:39:28.408143 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:28.408107 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1acf0f40-6f96-4914-8350-c5e19015968c-maas-api-xks-ca" pod "1acf0f40-6f96-4914-8350-c5e19015968c" (UID: "1acf0f40-6f96-4914-8350-c5e19015968c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:39:28.408332 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:28.408309 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1acf0f40-6f96-4914-8350-c5e19015968c-kube-api-access-86cj7" pod "1acf0f40-6f96-4914-8350-c5e19015968c" (UID: "1acf0f40-6f96-4914-8350-c5e19015968c"). InnerVolumeSpecName "kube-api-access-86cj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:39:28.491635 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:28.491598 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd" Oct 30 21:39:28.492042 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:28.492016 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd" Oct 30 21:39:28.493002 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:28.492980 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd" Oct 30 21:39:28.508153 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:28.508124 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1acf0f40-6f96-4914-8350-c5e19015968c-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:28.508310 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:28.508156 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86cj7\" (UniqueName: \"kubernetes.io/projected/1acf0f40-6f96-4914-8350-c5e19015968c-kube-api-access-86cj7\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:28.508310 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:28.508173 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1acf0f40-6f96-4914-8350-c5e19015968c-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:29.105001 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:29.104972 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b425e42-2fnd9" Oct 30 21:39:29.105216 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:29.105007 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b425e42-2fnd9" event={"ID":"6b6e60cc-ea7d-4c1e-bef8-3178637efa0b","Type":"ContainerDied","Data":"dcec7350a46dedf678a0190a1b618c06fa7e9f0d4a654eb5f18a5e125e48d4d2"} Oct 30 21:39:29.105216 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:29.105061 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dcec7350a46dedf678a0190a1b618c06fa7e9f0d4a654eb5f18a5e125e48d4d2" Oct 30 21:39:29.106497 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:29.106469 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd95a157-qmjvl" Oct 30 21:39:29.106627 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:29.106529 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd95a157-qmjvl" event={"ID":"1acf0f40-6f96-4914-8350-c5e19015968c","Type":"ContainerDied","Data":"4a7aeb085b854f40a3b4d79d93de736b10498449617bf9159bf9851c728f8e67"} Oct 30 21:39:29.106627 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:29.106581 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a7aeb085b854f40a3b4d79d93de736b10498449617bf9159bf9851c728f8e67" Oct 30 21:39:30.144774 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:39:30.144737 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:39:32.726724 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:32.726690 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5758a8e8-2s9zv"] Oct 30 21:39:32.727281 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:32.727261 5748 cpu_manager.go:414] "containerMap: removing container" podUID="1acf0f40-6f96-4914-8350-c5e19015968c" containerName="revoke-keys" Oct 30 21:39:32.727336 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:32.727286 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1acf0f40-6f96-4914-8350-c5e19015968c" containerName="revoke-keys" Oct 30 21:39:32.727373 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:32.727341 5748 cpu_manager.go:414] "containerMap: removing container" podUID="6b6e60cc-ea7d-4c1e-bef8-3178637efa0b" containerName="revoke-keys" Oct 30 21:39:32.727373 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:32.727348 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b6e60cc-ea7d-4c1e-bef8-3178637efa0b" containerName="revoke-keys" Oct 30 21:39:32.727443 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:32.727423 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b6e60cc-ea7d-4c1e-bef8-3178637efa0b" containerName="revoke-keys" Oct 30 21:39:32.727443 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:32.727434 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="1acf0f40-6f96-4914-8350-c5e19015968c" containerName="revoke-keys" Oct 30 21:39:32.731004 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:32.730989 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5758a8e8-2s9zv" Oct 30 21:39:32.748213 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:32.748179 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd1c21b8-d727-4ab8-9260-69a7fb3340c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5758a8e8-2s9zv\" (UID: \"fd1c21b8-d727-4ab8-9260-69a7fb3340c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5758a8e8-2s9zv" Oct 30 21:39:32.748333 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:32.748289 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd1c21b8-d727-4ab8-9260-69a7fb3340c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5758a8e8-2s9zv\" (UID: \"fd1c21b8-d727-4ab8-9260-69a7fb3340c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5758a8e8-2s9zv" Oct 30 21:39:32.748379 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:32.748346 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6sv8\" (UniqueName: \"kubernetes.io/projected/fd1c21b8-d727-4ab8-9260-69a7fb3340c0-kube-api-access-g6sv8\") pod \"maas-api-revoke-sub-5758a8e8-2s9zv\" (UID: \"fd1c21b8-d727-4ab8-9260-69a7fb3340c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5758a8e8-2s9zv" Oct 30 21:39:32.836136 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:32.836096 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:39:32.836321 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:32.836139 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:39:32.836321 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:32.836164 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:39:32.836321 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:32.836258 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 21:39:32.838654 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:32.838631 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5758a8e8-2s9zv"] Oct 30 21:39:32.849325 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:32.849301 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd1c21b8-d727-4ab8-9260-69a7fb3340c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5758a8e8-2s9zv\" (UID: \"fd1c21b8-d727-4ab8-9260-69a7fb3340c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5758a8e8-2s9zv" Oct 30 21:39:32.849462 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:32.849395 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd1c21b8-d727-4ab8-9260-69a7fb3340c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5758a8e8-2s9zv\" (UID: \"fd1c21b8-d727-4ab8-9260-69a7fb3340c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5758a8e8-2s9zv" Oct 30 21:39:32.849462 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:32.849446 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6sv8\" (UniqueName: \"kubernetes.io/projected/fd1c21b8-d727-4ab8-9260-69a7fb3340c0-kube-api-access-g6sv8\") pod \"maas-api-revoke-sub-5758a8e8-2s9zv\" (UID: \"fd1c21b8-d727-4ab8-9260-69a7fb3340c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5758a8e8-2s9zv" Oct 30 21:39:32.850033 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:32.850013 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd1c21b8-d727-4ab8-9260-69a7fb3340c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5758a8e8-2s9zv\" (UID: \"fd1c21b8-d727-4ab8-9260-69a7fb3340c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5758a8e8-2s9zv" Oct 30 21:39:32.851712 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:32.851684 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd1c21b8-d727-4ab8-9260-69a7fb3340c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5758a8e8-2s9zv\" (UID: \"fd1c21b8-d727-4ab8-9260-69a7fb3340c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5758a8e8-2s9zv" Oct 30 21:39:32.911083 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:32.911048 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6sv8\" (UniqueName: \"kubernetes.io/projected/fd1c21b8-d727-4ab8-9260-69a7fb3340c0-kube-api-access-g6sv8\") pod \"maas-api-revoke-sub-5758a8e8-2s9zv\" (UID: \"fd1c21b8-d727-4ab8-9260-69a7fb3340c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5758a8e8-2s9zv" Oct 30 21:39:33.043509 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:33.043418 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5758a8e8-2s9zv" Oct 30 21:39:33.426986 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:33.426951 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5758a8e8-2s9zv"] Oct 30 21:39:33.428832 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:39:33.428806 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd1c21b8_d727_4ab8_9260_69a7fb3340c0.slice/crio-bf8fd95d41bb5205730e36d882c20e4abea04336eafb6fefb269e738a375c612 WatchSource:0}: Error finding container bf8fd95d41bb5205730e36d882c20e4abea04336eafb6fefb269e738a375c612: Status 404 returned error can't find the container with id bf8fd95d41bb5205730e36d882c20e4abea04336eafb6fefb269e738a375c612 Oct 30 21:39:34.139938 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:34.139875 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5758a8e8-2s9zv" event={"ID":"fd1c21b8-d727-4ab8-9260-69a7fb3340c0","Type":"ContainerStarted","Data":"28e43d94a32926571dbc160ea478ba5200a9ea8a164cf0ec0b1f8c0f7af4ae5f"} Oct 30 21:39:34.140324 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:34.139959 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5758a8e8-2s9zv" event={"ID":"fd1c21b8-d727-4ab8-9260-69a7fb3340c0","Type":"ContainerStarted","Data":"bf8fd95d41bb5205730e36d882c20e4abea04336eafb6fefb269e738a375c612"} Oct 30 21:39:34.259364 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:34.259295 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5758a8e8-2s9zv" podStartSLOduration=2.259273519 podStartE2EDuration="2.259273519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:34.257673972 +0000 UTC m=+958.000194908" watchObservedRunningTime="2026-10-30 21:39:34.259273519 +0000 UTC m=+958.001794455" Oct 30 21:39:35.148474 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:35.148436 5748 generic.go:356] "Generic (PLEG): container finished" podID="fd1c21b8-d727-4ab8-9260-69a7fb3340c0" containerID="28e43d94a32926571dbc160ea478ba5200a9ea8a164cf0ec0b1f8c0f7af4ae5f" exitCode=0 Oct 30 21:39:35.148922 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:35.148501 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5758a8e8-2s9zv" event={"ID":"fd1c21b8-d727-4ab8-9260-69a7fb3340c0","Type":"ContainerDied","Data":"28e43d94a32926571dbc160ea478ba5200a9ea8a164cf0ec0b1f8c0f7af4ae5f"} Oct 30 21:39:36.292845 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:36.292819 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5758a8e8-2s9zv" Oct 30 21:39:36.385513 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:36.385476 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd1c21b8-d727-4ab8-9260-69a7fb3340c0-kube-api-access-g6sv8\" (UniqueName: \"kubernetes.io/projected/fd1c21b8-d727-4ab8-9260-69a7fb3340c0-kube-api-access-g6sv8\") pod \"fd1c21b8-d727-4ab8-9260-69a7fb3340c0\" (UID: \"fd1c21b8-d727-4ab8-9260-69a7fb3340c0\") " Oct 30 21:39:36.385685 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:36.385547 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd1c21b8-d727-4ab8-9260-69a7fb3340c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd1c21b8-d727-4ab8-9260-69a7fb3340c0-maas-api-openshift-ca\") pod \"fd1c21b8-d727-4ab8-9260-69a7fb3340c0\" (UID: \"fd1c21b8-d727-4ab8-9260-69a7fb3340c0\") " Oct 30 21:39:36.385685 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:36.385604 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd1c21b8-d727-4ab8-9260-69a7fb3340c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd1c21b8-d727-4ab8-9260-69a7fb3340c0-maas-api-xks-ca\") pod \"fd1c21b8-d727-4ab8-9260-69a7fb3340c0\" (UID: \"fd1c21b8-d727-4ab8-9260-69a7fb3340c0\") " Oct 30 21:39:36.385956 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:36.385928 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd1c21b8-d727-4ab8-9260-69a7fb3340c0-maas-api-openshift-ca" pod "fd1c21b8-d727-4ab8-9260-69a7fb3340c0" (UID: "fd1c21b8-d727-4ab8-9260-69a7fb3340c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:39:36.387714 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:36.387687 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd1c21b8-d727-4ab8-9260-69a7fb3340c0-kube-api-access-g6sv8" pod "fd1c21b8-d727-4ab8-9260-69a7fb3340c0" (UID: "fd1c21b8-d727-4ab8-9260-69a7fb3340c0"). InnerVolumeSpecName "kube-api-access-g6sv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:39:36.387810 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:36.387725 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd1c21b8-d727-4ab8-9260-69a7fb3340c0-maas-api-xks-ca" pod "fd1c21b8-d727-4ab8-9260-69a7fb3340c0" (UID: "fd1c21b8-d727-4ab8-9260-69a7fb3340c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:39:36.486483 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:36.486451 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6sv8\" (UniqueName: \"kubernetes.io/projected/fd1c21b8-d727-4ab8-9260-69a7fb3340c0-kube-api-access-g6sv8\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:36.486483 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:36.486481 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd1c21b8-d727-4ab8-9260-69a7fb3340c0-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:36.486483 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:36.486491 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd1c21b8-d727-4ab8-9260-69a7fb3340c0-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:37.161769 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:37.161736 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5758a8e8-2s9zv" Oct 30 21:39:37.161975 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:37.161784 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5758a8e8-2s9zv" event={"ID":"fd1c21b8-d727-4ab8-9260-69a7fb3340c0","Type":"ContainerDied","Data":"bf8fd95d41bb5205730e36d882c20e4abea04336eafb6fefb269e738a375c612"} Oct 30 21:39:37.161975 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:37.161878 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf8fd95d41bb5205730e36d882c20e4abea04336eafb6fefb269e738a375c612" Oct 30 21:39:43.611556 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:43.611512 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85b4ac7a-g9rn7"] Oct 30 21:39:43.612078 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:43.612061 5748 cpu_manager.go:414] "containerMap: removing container" podUID="fd1c21b8-d727-4ab8-9260-69a7fb3340c0" containerName="revoke-keys" Oct 30 21:39:43.612125 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:43.612079 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd1c21b8-d727-4ab8-9260-69a7fb3340c0" containerName="revoke-keys" Oct 30 21:39:43.612201 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:43.612191 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd1c21b8-d727-4ab8-9260-69a7fb3340c0" containerName="revoke-keys" Oct 30 21:39:43.616716 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:43.616700 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b4ac7a-g9rn7" Oct 30 21:39:43.660578 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:43.660539 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/536af565-9de6-4203-b850-39ec9ab23315-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85b4ac7a-g9rn7\" (UID: \"536af565-9de6-4203-b850-39ec9ab23315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b4ac7a-g9rn7" Oct 30 21:39:43.660761 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:43.660656 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgdhl\" (UniqueName: \"kubernetes.io/projected/536af565-9de6-4203-b850-39ec9ab23315-kube-api-access-bgdhl\") pod \"maas-api-revoke-sub-85b4ac7a-g9rn7\" (UID: \"536af565-9de6-4203-b850-39ec9ab23315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b4ac7a-g9rn7" Oct 30 21:39:43.660801 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:43.660778 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/536af565-9de6-4203-b850-39ec9ab23315-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85b4ac7a-g9rn7\" (UID: \"536af565-9de6-4203-b850-39ec9ab23315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b4ac7a-g9rn7" Oct 30 21:39:43.673430 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:43.673390 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85b4ac7a-g9rn7"] Oct 30 21:39:43.720534 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:43.720497 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:39:43.720534 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:43.720536 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 21:39:43.720734 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:43.720568 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:39:43.720734 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:43.720615 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:39:43.762094 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:43.762056 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/536af565-9de6-4203-b850-39ec9ab23315-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85b4ac7a-g9rn7\" (UID: \"536af565-9de6-4203-b850-39ec9ab23315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b4ac7a-g9rn7" Oct 30 21:39:43.762271 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:43.762132 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgdhl\" (UniqueName: \"kubernetes.io/projected/536af565-9de6-4203-b850-39ec9ab23315-kube-api-access-bgdhl\") pod \"maas-api-revoke-sub-85b4ac7a-g9rn7\" (UID: \"536af565-9de6-4203-b850-39ec9ab23315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b4ac7a-g9rn7" Oct 30 21:39:43.762271 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:43.762225 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/536af565-9de6-4203-b850-39ec9ab23315-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85b4ac7a-g9rn7\" (UID: \"536af565-9de6-4203-b850-39ec9ab23315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b4ac7a-g9rn7" Oct 30 21:39:43.762922 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:43.762875 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/536af565-9de6-4203-b850-39ec9ab23315-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85b4ac7a-g9rn7\" (UID: \"536af565-9de6-4203-b850-39ec9ab23315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b4ac7a-g9rn7" Oct 30 21:39:43.764526 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:43.764506 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/536af565-9de6-4203-b850-39ec9ab23315-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85b4ac7a-g9rn7\" (UID: \"536af565-9de6-4203-b850-39ec9ab23315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b4ac7a-g9rn7" Oct 30 21:39:43.824108 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:43.824070 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgdhl\" (UniqueName: \"kubernetes.io/projected/536af565-9de6-4203-b850-39ec9ab23315-kube-api-access-bgdhl\") pod \"maas-api-revoke-sub-85b4ac7a-g9rn7\" (UID: \"536af565-9de6-4203-b850-39ec9ab23315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b4ac7a-g9rn7" Oct 30 21:39:43.929712 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:43.929675 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b4ac7a-g9rn7" Oct 30 21:39:44.308682 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:44.308657 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85b4ac7a-g9rn7"] Oct 30 21:39:44.310994 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:39:44.310960 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod536af565_9de6_4203_b850_39ec9ab23315.slice/crio-7de5de80d0cf7130735379227b2f0e639fb1d6ad8b3428f663ab0836fb17b3ad WatchSource:0}: Error finding container 7de5de80d0cf7130735379227b2f0e639fb1d6ad8b3428f663ab0836fb17b3ad: Status 404 returned error can't find the container with id 7de5de80d0cf7130735379227b2f0e639fb1d6ad8b3428f663ab0836fb17b3ad Oct 30 21:39:44.463170 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:44.463137 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w9njw"] Oct 30 21:39:44.468380 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:44.468356 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w9njw" Oct 30 21:39:44.567818 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:44.567744 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w9njw"] Oct 30 21:39:44.571508 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:44.571468 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eddd7c6a-a409-468c-831a-2e3944e8059a-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-w9njw\" (UID: \"eddd7c6a-a409-468c-831a-2e3944e8059a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w9njw" Oct 30 21:39:44.571706 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:44.571520 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eddd7c6a-a409-468c-831a-2e3944e8059a-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-w9njw\" (UID: \"eddd7c6a-a409-468c-831a-2e3944e8059a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w9njw" Oct 30 21:39:44.571706 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:44.571637 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eddd7c6a-a409-468c-831a-2e3944e8059a-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-w9njw\" (UID: \"eddd7c6a-a409-468c-831a-2e3944e8059a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w9njw" Oct 30 21:39:44.571706 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:44.571687 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eddd7c6a-a409-468c-831a-2e3944e8059a-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-w9njw\" (UID: \"eddd7c6a-a409-468c-831a-2e3944e8059a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w9njw" Oct 30 21:39:44.571886 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:44.571730 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eddd7c6a-a409-468c-831a-2e3944e8059a-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-w9njw\" (UID: \"eddd7c6a-a409-468c-831a-2e3944e8059a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w9njw" Oct 30 21:39:44.571886 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:44.571769 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrbvm\" (UniqueName: \"kubernetes.io/projected/eddd7c6a-a409-468c-831a-2e3944e8059a-kube-api-access-hrbvm\") pod \"maas-default-gateway-openshift-default-75d7c785f-w9njw\" (UID: \"eddd7c6a-a409-468c-831a-2e3944e8059a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w9njw" Oct 30 21:39:44.571886 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:44.571822 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eddd7c6a-a409-468c-831a-2e3944e8059a-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-w9njw\" (UID: \"eddd7c6a-a409-468c-831a-2e3944e8059a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w9njw" Oct 30 21:39:44.572052 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:44.571939 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eddd7c6a-a409-468c-831a-2e3944e8059a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-w9njw\" (UID: \"eddd7c6a-a409-468c-831a-2e3944e8059a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w9njw" Oct 30 21:39:44.572052 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:44.571966 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eddd7c6a-a409-468c-831a-2e3944e8059a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-w9njw\" (UID: \"eddd7c6a-a409-468c-831a-2e3944e8059a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w9njw" Oct 30 21:39:44.673559 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:44.673528 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eddd7c6a-a409-468c-831a-2e3944e8059a-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-w9njw\" (UID: \"eddd7c6a-a409-468c-831a-2e3944e8059a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w9njw" Oct 30 21:39:44.673559 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:44.673560 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eddd7c6a-a409-468c-831a-2e3944e8059a-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-w9njw\" (UID: \"eddd7c6a-a409-468c-831a-2e3944e8059a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w9njw" Oct 30 21:39:44.674024 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:44.673593 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eddd7c6a-a409-468c-831a-2e3944e8059a-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-w9njw\" (UID: \"eddd7c6a-a409-468c-831a-2e3944e8059a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w9njw" Oct 30 21:39:44.674024 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:44.673771 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eddd7c6a-a409-468c-831a-2e3944e8059a-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-w9njw\" (UID: \"eddd7c6a-a409-468c-831a-2e3944e8059a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w9njw" Oct 30 21:39:44.674024 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:44.673831 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eddd7c6a-a409-468c-831a-2e3944e8059a-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-w9njw\" (UID: \"eddd7c6a-a409-468c-831a-2e3944e8059a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w9njw" Oct 30 21:39:44.674024 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:44.673869 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrbvm\" (UniqueName: \"kubernetes.io/projected/eddd7c6a-a409-468c-831a-2e3944e8059a-kube-api-access-hrbvm\") pod \"maas-default-gateway-openshift-default-75d7c785f-w9njw\" (UID: \"eddd7c6a-a409-468c-831a-2e3944e8059a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w9njw" Oct 30 21:39:44.674024 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:44.673958 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eddd7c6a-a409-468c-831a-2e3944e8059a-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-w9njw\" (UID: \"eddd7c6a-a409-468c-831a-2e3944e8059a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w9njw" Oct 30 21:39:44.674024 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:44.673991 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eddd7c6a-a409-468c-831a-2e3944e8059a-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-w9njw\" (UID: \"eddd7c6a-a409-468c-831a-2e3944e8059a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w9njw" Oct 30 21:39:44.674325 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:44.674098 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eddd7c6a-a409-468c-831a-2e3944e8059a-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-w9njw\" (UID: \"eddd7c6a-a409-468c-831a-2e3944e8059a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w9njw" Oct 30 21:39:44.674325 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:44.674151 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eddd7c6a-a409-468c-831a-2e3944e8059a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-w9njw\" (UID: \"eddd7c6a-a409-468c-831a-2e3944e8059a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w9njw" Oct 30 21:39:44.674325 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:44.674177 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eddd7c6a-a409-468c-831a-2e3944e8059a-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-w9njw\" (UID: \"eddd7c6a-a409-468c-831a-2e3944e8059a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w9njw" Oct 30 21:39:44.674325 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:44.674181 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eddd7c6a-a409-468c-831a-2e3944e8059a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-w9njw\" (UID: \"eddd7c6a-a409-468c-831a-2e3944e8059a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w9njw" Oct 30 21:39:44.674325 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:44.674275 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eddd7c6a-a409-468c-831a-2e3944e8059a-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-w9njw\" (UID: \"eddd7c6a-a409-468c-831a-2e3944e8059a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w9njw" Oct 30 21:39:44.674768 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:44.674744 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eddd7c6a-a409-468c-831a-2e3944e8059a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-w9njw\" (UID: \"eddd7c6a-a409-468c-831a-2e3944e8059a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w9njw" Oct 30 21:39:44.675837 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:44.675815 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eddd7c6a-a409-468c-831a-2e3944e8059a-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-w9njw\" (UID: \"eddd7c6a-a409-468c-831a-2e3944e8059a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w9njw" Oct 30 21:39:44.676392 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:44.676376 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eddd7c6a-a409-468c-831a-2e3944e8059a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-w9njw\" (UID: \"eddd7c6a-a409-468c-831a-2e3944e8059a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w9njw" Oct 30 21:39:44.735949 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:44.735909 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eddd7c6a-a409-468c-831a-2e3944e8059a-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-w9njw\" (UID: \"eddd7c6a-a409-468c-831a-2e3944e8059a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w9njw" Oct 30 21:39:44.736076 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:44.736047 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrbvm\" (UniqueName: \"kubernetes.io/projected/eddd7c6a-a409-468c-831a-2e3944e8059a-kube-api-access-hrbvm\") pod \"maas-default-gateway-openshift-default-75d7c785f-w9njw\" (UID: \"eddd7c6a-a409-468c-831a-2e3944e8059a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w9njw" Oct 30 21:39:44.782015 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:44.781981 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w9njw" Oct 30 21:39:44.855400 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:44.855367 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-796e1be4-cf4mn"] Oct 30 21:39:44.860186 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:44.860165 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-796e1be4-cf4mn" Oct 30 21:39:44.908960 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:44.908926 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-796e1be4-cf4mn"] Oct 30 21:39:44.978536 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:44.978494 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad271c61-4a3c-4432-baed-2e4f5045c471-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-796e1be4-cf4mn\" (UID: \"ad271c61-4a3c-4432-baed-2e4f5045c471\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-796e1be4-cf4mn" Oct 30 21:39:44.978697 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:44.978606 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cffsq\" (UniqueName: \"kubernetes.io/projected/ad271c61-4a3c-4432-baed-2e4f5045c471-kube-api-access-cffsq\") pod \"maas-api-revoke-sub-796e1be4-cf4mn\" (UID: \"ad271c61-4a3c-4432-baed-2e4f5045c471\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-796e1be4-cf4mn" Oct 30 21:39:44.978749 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:44.978734 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad271c61-4a3c-4432-baed-2e4f5045c471-maas-api-xks-ca\") pod \"maas-api-revoke-sub-796e1be4-cf4mn\" (UID: \"ad271c61-4a3c-4432-baed-2e4f5045c471\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-796e1be4-cf4mn" Oct 30 21:39:45.080146 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:45.080110 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad271c61-4a3c-4432-baed-2e4f5045c471-maas-api-xks-ca\") pod \"maas-api-revoke-sub-796e1be4-cf4mn\" (UID: \"ad271c61-4a3c-4432-baed-2e4f5045c471\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-796e1be4-cf4mn" Oct 30 21:39:45.080309 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:45.080169 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad271c61-4a3c-4432-baed-2e4f5045c471-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-796e1be4-cf4mn\" (UID: \"ad271c61-4a3c-4432-baed-2e4f5045c471\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-796e1be4-cf4mn" Oct 30 21:39:45.080309 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:45.080223 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cffsq\" (UniqueName: \"kubernetes.io/projected/ad271c61-4a3c-4432-baed-2e4f5045c471-kube-api-access-cffsq\") pod \"maas-api-revoke-sub-796e1be4-cf4mn\" (UID: \"ad271c61-4a3c-4432-baed-2e4f5045c471\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-796e1be4-cf4mn" Oct 30 21:39:45.080801 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:45.080777 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad271c61-4a3c-4432-baed-2e4f5045c471-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-796e1be4-cf4mn\" (UID: \"ad271c61-4a3c-4432-baed-2e4f5045c471\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-796e1be4-cf4mn" Oct 30 21:39:45.082398 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:45.082376 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad271c61-4a3c-4432-baed-2e4f5045c471-maas-api-xks-ca\") pod \"maas-api-revoke-sub-796e1be4-cf4mn\" (UID: \"ad271c61-4a3c-4432-baed-2e4f5045c471\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-796e1be4-cf4mn" Oct 30 21:39:45.141687 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:45.141653 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cffsq\" (UniqueName: \"kubernetes.io/projected/ad271c61-4a3c-4432-baed-2e4f5045c471-kube-api-access-cffsq\") pod \"maas-api-revoke-sub-796e1be4-cf4mn\" (UID: \"ad271c61-4a3c-4432-baed-2e4f5045c471\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-796e1be4-cf4mn" Oct 30 21:39:45.166460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:45.166431 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w9njw"] Oct 30 21:39:45.167389 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:39:45.167355 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeddd7c6a_a409_468c_831a_2e3944e8059a.slice/crio-3818df89a2954358a1a830aaf348b9ddf1b467e0f42677d93526e4fa75e33431 WatchSource:0}: Error finding container 3818df89a2954358a1a830aaf348b9ddf1b467e0f42677d93526e4fa75e33431: Status 404 returned error can't find the container with id 3818df89a2954358a1a830aaf348b9ddf1b467e0f42677d93526e4fa75e33431 Oct 30 21:39:45.169838 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:45.169807 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:39:45.169946 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:45.169871 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:39:45.169946 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:45.169925 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:39:45.174097 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:45.174076 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-796e1be4-cf4mn" Oct 30 21:39:45.209335 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:45.209303 5748 generic.go:356] "Generic (PLEG): container finished" podID="536af565-9de6-4203-b850-39ec9ab23315" containerID="5e9071b82ec895db26e20b5edf4ec7213468e4147a31768d0b9158e3fd784053" exitCode=0 Oct 30 21:39:45.209473 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:45.209345 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b4ac7a-g9rn7" event={"ID":"536af565-9de6-4203-b850-39ec9ab23315","Type":"ContainerDied","Data":"5e9071b82ec895db26e20b5edf4ec7213468e4147a31768d0b9158e3fd784053"} Oct 30 21:39:45.209473 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:45.209421 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b4ac7a-g9rn7" event={"ID":"536af565-9de6-4203-b850-39ec9ab23315","Type":"ContainerStarted","Data":"7de5de80d0cf7130735379227b2f0e639fb1d6ad8b3428f663ab0836fb17b3ad"} Oct 30 21:39:45.210766 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:45.210743 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w9njw" event={"ID":"eddd7c6a-a409-468c-831a-2e3944e8059a","Type":"ContainerStarted","Data":"3818df89a2954358a1a830aaf348b9ddf1b467e0f42677d93526e4fa75e33431"} Oct 30 21:39:45.554930 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:45.554883 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-796e1be4-cf4mn"] Oct 30 21:39:45.556364 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:39:45.556333 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad271c61_4a3c_4432_baed_2e4f5045c471.slice/crio-f71aa10251c449d953a7dd16387f8824a1ad34c3672f62d96ef2c9f425b00de8 WatchSource:0}: Error finding container f71aa10251c449d953a7dd16387f8824a1ad34c3672f62d96ef2c9f425b00de8: Status 404 returned error can't find the container with id f71aa10251c449d953a7dd16387f8824a1ad34c3672f62d96ef2c9f425b00de8 Oct 30 21:39:46.235035 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:46.234943 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w9njw" event={"ID":"eddd7c6a-a409-468c-831a-2e3944e8059a","Type":"ContainerStarted","Data":"28e30ab5b8b860f6025b5609190944393492289c5c824f209d025908dc449e5a"} Oct 30 21:39:46.236675 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:46.236647 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-796e1be4-cf4mn" event={"ID":"ad271c61-4a3c-4432-baed-2e4f5045c471","Type":"ContainerStarted","Data":"6b9d86defbfb58f739f12842e5b15c7e0e3e20e6b3be40f58de94a53c29a5ec8"} Oct 30 21:39:46.236811 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:46.236687 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-796e1be4-cf4mn" event={"ID":"ad271c61-4a3c-4432-baed-2e4f5045c471","Type":"ContainerStarted","Data":"f71aa10251c449d953a7dd16387f8824a1ad34c3672f62d96ef2c9f425b00de8"} Oct 30 21:39:46.358028 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:46.357973 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w9njw" podStartSLOduration=2.35795805 podStartE2EDuration="2.35795805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:46.356369322 +0000 UTC m=+970.098890258" watchObservedRunningTime="2026-10-30 21:39:46.35795805 +0000 UTC m=+970.100478984" Oct 30 21:39:46.399736 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:46.399710 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b4ac7a-g9rn7" Oct 30 21:39:46.484533 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:46.484479 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-796e1be4-cf4mn" podStartSLOduration=2.484465046 podStartE2EDuration="2.484465046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:46.481781493 +0000 UTC m=+970.224302429" watchObservedRunningTime="2026-10-30 21:39:46.484465046 +0000 UTC m=+970.226986006" Oct 30 21:39:46.496508 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:46.496435 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/536af565-9de6-4203-b850-39ec9ab23315-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/536af565-9de6-4203-b850-39ec9ab23315-maas-api-openshift-ca\") pod \"536af565-9de6-4203-b850-39ec9ab23315\" (UID: \"536af565-9de6-4203-b850-39ec9ab23315\") " Oct 30 21:39:46.496508 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:46.496476 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/536af565-9de6-4203-b850-39ec9ab23315-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/536af565-9de6-4203-b850-39ec9ab23315-maas-api-xks-ca\") pod \"536af565-9de6-4203-b850-39ec9ab23315\" (UID: \"536af565-9de6-4203-b850-39ec9ab23315\") " Oct 30 21:39:46.496678 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:46.496561 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/536af565-9de6-4203-b850-39ec9ab23315-kube-api-access-bgdhl\" (UniqueName: \"kubernetes.io/projected/536af565-9de6-4203-b850-39ec9ab23315-kube-api-access-bgdhl\") pod \"536af565-9de6-4203-b850-39ec9ab23315\" (UID: \"536af565-9de6-4203-b850-39ec9ab23315\") " Oct 30 21:39:46.496813 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:46.496783 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/536af565-9de6-4203-b850-39ec9ab23315-maas-api-openshift-ca" pod "536af565-9de6-4203-b850-39ec9ab23315" (UID: "536af565-9de6-4203-b850-39ec9ab23315"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:39:46.497059 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:46.497032 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/536af565-9de6-4203-b850-39ec9ab23315-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:46.498710 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:46.498683 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/536af565-9de6-4203-b850-39ec9ab23315-maas-api-xks-ca" pod "536af565-9de6-4203-b850-39ec9ab23315" (UID: "536af565-9de6-4203-b850-39ec9ab23315"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:39:46.498797 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:46.498730 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/536af565-9de6-4203-b850-39ec9ab23315-kube-api-access-bgdhl" pod "536af565-9de6-4203-b850-39ec9ab23315" (UID: "536af565-9de6-4203-b850-39ec9ab23315"). InnerVolumeSpecName "kube-api-access-bgdhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:39:46.598259 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:46.598216 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/536af565-9de6-4203-b850-39ec9ab23315-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:46.598259 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:46.598254 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgdhl\" (UniqueName: \"kubernetes.io/projected/536af565-9de6-4203-b850-39ec9ab23315-kube-api-access-bgdhl\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:46.783036 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:46.783001 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w9njw" Oct 30 21:39:46.784653 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:46.784627 5748 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-w9njw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 21:39:46.784766 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:46.784675 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w9njw" podUID="eddd7c6a-a409-468c-831a-2e3944e8059a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 21:39:47.243915 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:47.243868 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b4ac7a-g9rn7" Oct 30 21:39:47.244331 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:47.243889 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b4ac7a-g9rn7" event={"ID":"536af565-9de6-4203-b850-39ec9ab23315","Type":"ContainerDied","Data":"7de5de80d0cf7130735379227b2f0e639fb1d6ad8b3428f663ab0836fb17b3ad"} Oct 30 21:39:47.244331 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:47.243970 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7de5de80d0cf7130735379227b2f0e639fb1d6ad8b3428f663ab0836fb17b3ad" Oct 30 21:39:47.245608 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:47.245582 5748 generic.go:356] "Generic (PLEG): container finished" podID="ad271c61-4a3c-4432-baed-2e4f5045c471" containerID="6b9d86defbfb58f739f12842e5b15c7e0e3e20e6b3be40f58de94a53c29a5ec8" exitCode=0 Oct 30 21:39:47.245749 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:47.245627 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-796e1be4-cf4mn" event={"ID":"ad271c61-4a3c-4432-baed-2e4f5045c471","Type":"ContainerDied","Data":"6b9d86defbfb58f739f12842e5b15c7e0e3e20e6b3be40f58de94a53c29a5ec8"} Oct 30 21:39:47.782685 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:47.782653 5748 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-w9njw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 21:39:47.782863 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:47.782706 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w9njw" podUID="eddd7c6a-a409-468c-831a-2e3944e8059a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 21:39:48.450454 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:48.450427 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-796e1be4-cf4mn" Oct 30 21:39:48.544494 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:48.544407 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad271c61-4a3c-4432-baed-2e4f5045c471-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad271c61-4a3c-4432-baed-2e4f5045c471-maas-api-openshift-ca\") pod \"ad271c61-4a3c-4432-baed-2e4f5045c471\" (UID: \"ad271c61-4a3c-4432-baed-2e4f5045c471\") " Oct 30 21:39:48.544494 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:48.544476 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad271c61-4a3c-4432-baed-2e4f5045c471-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad271c61-4a3c-4432-baed-2e4f5045c471-maas-api-xks-ca\") pod \"ad271c61-4a3c-4432-baed-2e4f5045c471\" (UID: \"ad271c61-4a3c-4432-baed-2e4f5045c471\") " Oct 30 21:39:48.544693 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:48.544551 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad271c61-4a3c-4432-baed-2e4f5045c471-kube-api-access-cffsq\" (UniqueName: \"kubernetes.io/projected/ad271c61-4a3c-4432-baed-2e4f5045c471-kube-api-access-cffsq\") pod \"ad271c61-4a3c-4432-baed-2e4f5045c471\" (UID: \"ad271c61-4a3c-4432-baed-2e4f5045c471\") " Oct 30 21:39:48.544807 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:48.544765 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad271c61-4a3c-4432-baed-2e4f5045c471-maas-api-openshift-ca" pod "ad271c61-4a3c-4432-baed-2e4f5045c471" (UID: "ad271c61-4a3c-4432-baed-2e4f5045c471"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:39:48.545005 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:48.544981 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad271c61-4a3c-4432-baed-2e4f5045c471-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:48.546750 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:48.546729 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad271c61-4a3c-4432-baed-2e4f5045c471-maas-api-xks-ca" pod "ad271c61-4a3c-4432-baed-2e4f5045c471" (UID: "ad271c61-4a3c-4432-baed-2e4f5045c471"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:39:48.546829 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:48.546748 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad271c61-4a3c-4432-baed-2e4f5045c471-kube-api-access-cffsq" pod "ad271c61-4a3c-4432-baed-2e4f5045c471" (UID: "ad271c61-4a3c-4432-baed-2e4f5045c471"). InnerVolumeSpecName "kube-api-access-cffsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:39:48.646193 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:48.646154 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cffsq\" (UniqueName: \"kubernetes.io/projected/ad271c61-4a3c-4432-baed-2e4f5045c471-kube-api-access-cffsq\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:48.646193 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:48.646190 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad271c61-4a3c-4432-baed-2e4f5045c471-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:48.783286 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:48.783244 5748 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-w9njw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 21:39:48.783439 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:48.783308 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w9njw" podUID="eddd7c6a-a409-468c-831a-2e3944e8059a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 21:39:49.258684 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:49.258653 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-796e1be4-cf4mn" Oct 30 21:39:49.258863 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:49.258688 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-796e1be4-cf4mn" event={"ID":"ad271c61-4a3c-4432-baed-2e4f5045c471","Type":"ContainerDied","Data":"f71aa10251c449d953a7dd16387f8824a1ad34c3672f62d96ef2c9f425b00de8"} Oct 30 21:39:49.258863 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:49.258738 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f71aa10251c449d953a7dd16387f8824a1ad34c3672f62d96ef2c9f425b00de8" Oct 30 21:39:49.783231 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:49.783187 5748 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-w9njw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 21:39:49.783635 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:49.783248 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w9njw" podUID="eddd7c6a-a409-468c-831a-2e3944e8059a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 21:39:50.782835 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:50.782800 5748 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-w9njw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 21:39:50.783023 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:50.782921 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w9njw" podUID="eddd7c6a-a409-468c-831a-2e3944e8059a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 21:39:51.143359 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:51.143323 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5fe7552-mdv9r"] Oct 30 21:39:51.143818 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:51.143805 5748 cpu_manager.go:414] "containerMap: removing container" podUID="536af565-9de6-4203-b850-39ec9ab23315" containerName="revoke-keys" Oct 30 21:39:51.143860 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:51.143819 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="536af565-9de6-4203-b850-39ec9ab23315" containerName="revoke-keys" Oct 30 21:39:51.143860 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:51.143835 5748 cpu_manager.go:414] "containerMap: removing container" podUID="ad271c61-4a3c-4432-baed-2e4f5045c471" containerName="revoke-keys" Oct 30 21:39:51.143860 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:51.143840 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad271c61-4a3c-4432-baed-2e4f5045c471" containerName="revoke-keys" Oct 30 21:39:51.143991 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:51.143945 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="536af565-9de6-4203-b850-39ec9ab23315" containerName="revoke-keys" Oct 30 21:39:51.143991 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:51.143954 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad271c61-4a3c-4432-baed-2e4f5045c471" containerName="revoke-keys" Oct 30 21:39:51.147486 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:51.147463 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5fe7552-mdv9r" Oct 30 21:39:51.253992 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:51.253953 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:39:51.253992 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:51.253986 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 21:39:51.254290 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:51.253984 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:39:51.254290 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:51.253970 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:39:51.256782 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:51.256752 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5fe7552-mdv9r"] Oct 30 21:39:51.275544 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:51.275501 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c2cx\" (UniqueName: \"kubernetes.io/projected/559e85df-748d-447a-9adc-3975200f7895-kube-api-access-8c2cx\") pod \"maas-api-revoke-sub-a5fe7552-mdv9r\" (UID: \"559e85df-748d-447a-9adc-3975200f7895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5fe7552-mdv9r" Oct 30 21:39:51.275725 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:51.275566 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/559e85df-748d-447a-9adc-3975200f7895-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5fe7552-mdv9r\" (UID: \"559e85df-748d-447a-9adc-3975200f7895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5fe7552-mdv9r" Oct 30 21:39:51.275725 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:51.275643 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/559e85df-748d-447a-9adc-3975200f7895-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5fe7552-mdv9r\" (UID: \"559e85df-748d-447a-9adc-3975200f7895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5fe7552-mdv9r" Oct 30 21:39:51.377187 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:51.377143 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c2cx\" (UniqueName: \"kubernetes.io/projected/559e85df-748d-447a-9adc-3975200f7895-kube-api-access-8c2cx\") pod \"maas-api-revoke-sub-a5fe7552-mdv9r\" (UID: \"559e85df-748d-447a-9adc-3975200f7895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5fe7552-mdv9r" Oct 30 21:39:51.377366 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:51.377235 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/559e85df-748d-447a-9adc-3975200f7895-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5fe7552-mdv9r\" (UID: \"559e85df-748d-447a-9adc-3975200f7895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5fe7552-mdv9r" Oct 30 21:39:51.377366 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:51.377255 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/559e85df-748d-447a-9adc-3975200f7895-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5fe7552-mdv9r\" (UID: \"559e85df-748d-447a-9adc-3975200f7895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5fe7552-mdv9r" Oct 30 21:39:51.377954 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:51.377930 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/559e85df-748d-447a-9adc-3975200f7895-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5fe7552-mdv9r\" (UID: \"559e85df-748d-447a-9adc-3975200f7895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5fe7552-mdv9r" Oct 30 21:39:51.379774 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:51.379751 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/559e85df-748d-447a-9adc-3975200f7895-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5fe7552-mdv9r\" (UID: \"559e85df-748d-447a-9adc-3975200f7895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5fe7552-mdv9r" Oct 30 21:39:51.444843 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:51.444771 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c2cx\" (UniqueName: \"kubernetes.io/projected/559e85df-748d-447a-9adc-3975200f7895-kube-api-access-8c2cx\") pod \"maas-api-revoke-sub-a5fe7552-mdv9r\" (UID: \"559e85df-748d-447a-9adc-3975200f7895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5fe7552-mdv9r" Oct 30 21:39:51.461510 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:51.461470 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5fe7552-mdv9r" Oct 30 21:39:51.782697 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:51.782607 5748 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-w9njw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 21:39:51.782697 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:51.782675 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w9njw" podUID="eddd7c6a-a409-468c-831a-2e3944e8059a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 21:39:51.846971 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:51.846939 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5fe7552-mdv9r"] Oct 30 21:39:51.849726 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:39:51.849698 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod559e85df_748d_447a_9adc_3975200f7895.slice/crio-1c1cfc80ce2c957fa13e7c805333dbd741c1b0e2c9463886e008eb9f99f27541 WatchSource:0}: Error finding container 1c1cfc80ce2c957fa13e7c805333dbd741c1b0e2c9463886e008eb9f99f27541: Status 404 returned error can't find the container with id 1c1cfc80ce2c957fa13e7c805333dbd741c1b0e2c9463886e008eb9f99f27541 Oct 30 21:39:52.278513 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:52.278473 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5fe7552-mdv9r" event={"ID":"559e85df-748d-447a-9adc-3975200f7895","Type":"ContainerStarted","Data":"605068d78273e034e494ce8cef9bbb2519eec529dd7dd3de01802f777731c782"} Oct 30 21:39:52.279007 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:52.278543 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5fe7552-mdv9r" event={"ID":"559e85df-748d-447a-9adc-3975200f7895","Type":"ContainerStarted","Data":"1c1cfc80ce2c957fa13e7c805333dbd741c1b0e2c9463886e008eb9f99f27541"} Oct 30 21:39:52.398753 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:52.398690 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5fe7552-mdv9r" podStartSLOduration=1.3986714519999999 podStartE2EDuration="1.398671452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:52.394861776 +0000 UTC m=+976.137382711" watchObservedRunningTime="2026-10-30 21:39:52.398671452 +0000 UTC m=+976.141192389" Oct 30 21:39:52.783180 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:52.783140 5748 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-w9njw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 21:39:52.783339 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:52.783223 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w9njw" podUID="eddd7c6a-a409-468c-831a-2e3944e8059a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 21:39:53.289506 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:53.289468 5748 generic.go:356] "Generic (PLEG): container finished" podID="559e85df-748d-447a-9adc-3975200f7895" containerID="605068d78273e034e494ce8cef9bbb2519eec529dd7dd3de01802f777731c782" exitCode=0 Oct 30 21:39:53.289887 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:53.289513 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5fe7552-mdv9r" event={"ID":"559e85df-748d-447a-9adc-3975200f7895","Type":"ContainerDied","Data":"605068d78273e034e494ce8cef9bbb2519eec529dd7dd3de01802f777731c782"} Oct 30 21:39:53.787146 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:53.787113 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w9njw" Oct 30 21:39:53.787610 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:53.787592 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w9njw" Oct 30 21:39:53.788411 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:53.788392 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w9njw" Oct 30 21:39:54.440659 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:54.440632 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5fe7552-mdv9r" Oct 30 21:39:54.514337 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:54.514306 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/559e85df-748d-447a-9adc-3975200f7895-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/559e85df-748d-447a-9adc-3975200f7895-maas-api-xks-ca\") pod \"559e85df-748d-447a-9adc-3975200f7895\" (UID: \"559e85df-748d-447a-9adc-3975200f7895\") " Oct 30 21:39:54.514511 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:54.514374 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/559e85df-748d-447a-9adc-3975200f7895-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/559e85df-748d-447a-9adc-3975200f7895-maas-api-openshift-ca\") pod \"559e85df-748d-447a-9adc-3975200f7895\" (UID: \"559e85df-748d-447a-9adc-3975200f7895\") " Oct 30 21:39:54.514511 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:54.514400 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/559e85df-748d-447a-9adc-3975200f7895-kube-api-access-8c2cx\" (UniqueName: \"kubernetes.io/projected/559e85df-748d-447a-9adc-3975200f7895-kube-api-access-8c2cx\") pod \"559e85df-748d-447a-9adc-3975200f7895\" (UID: \"559e85df-748d-447a-9adc-3975200f7895\") " Oct 30 21:39:54.514726 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:54.514703 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/559e85df-748d-447a-9adc-3975200f7895-maas-api-openshift-ca" pod "559e85df-748d-447a-9adc-3975200f7895" (UID: "559e85df-748d-447a-9adc-3975200f7895"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:39:54.516398 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:54.516370 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/559e85df-748d-447a-9adc-3975200f7895-maas-api-xks-ca" pod "559e85df-748d-447a-9adc-3975200f7895" (UID: "559e85df-748d-447a-9adc-3975200f7895"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:39:54.516556 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:54.516537 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/559e85df-748d-447a-9adc-3975200f7895-kube-api-access-8c2cx" pod "559e85df-748d-447a-9adc-3975200f7895" (UID: "559e85df-748d-447a-9adc-3975200f7895"). InnerVolumeSpecName "kube-api-access-8c2cx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:39:54.615930 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:54.615824 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/559e85df-748d-447a-9adc-3975200f7895-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:54.615930 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:54.615853 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/559e85df-748d-447a-9adc-3975200f7895-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:54.615930 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:54.615865 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c2cx\" (UniqueName: \"kubernetes.io/projected/559e85df-748d-447a-9adc-3975200f7895-kube-api-access-8c2cx\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:55.302650 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:55.302620 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5fe7552-mdv9r" Oct 30 21:39:55.302800 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:55.302660 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5fe7552-mdv9r" event={"ID":"559e85df-748d-447a-9adc-3975200f7895","Type":"ContainerDied","Data":"1c1cfc80ce2c957fa13e7c805333dbd741c1b0e2c9463886e008eb9f99f27541"} Oct 30 21:39:55.302800 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:55.302707 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c1cfc80ce2c957fa13e7c805333dbd741c1b0e2c9463886e008eb9f99f27541" Oct 30 21:39:55.808243 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:55.808205 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7277fd6-62shf"] Oct 30 21:39:55.808929 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:55.808913 5748 cpu_manager.go:414] "containerMap: removing container" podUID="559e85df-748d-447a-9adc-3975200f7895" containerName="revoke-keys" Oct 30 21:39:55.808978 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:55.808931 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="559e85df-748d-447a-9adc-3975200f7895" containerName="revoke-keys" Oct 30 21:39:55.809098 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:55.809081 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="559e85df-748d-447a-9adc-3975200f7895" containerName="revoke-keys" Oct 30 21:39:55.812991 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:55.812973 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7277fd6-62shf" Oct 30 21:39:55.827967 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:55.827940 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8v8k\" (UniqueName: \"kubernetes.io/projected/1ea379a7-6815-4d5a-82ce-f87e0ae086c3-kube-api-access-x8v8k\") pod \"maas-api-revoke-sub-b7277fd6-62shf\" (UID: \"1ea379a7-6815-4d5a-82ce-f87e0ae086c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7277fd6-62shf" Oct 30 21:39:55.828077 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:55.827990 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ea379a7-6815-4d5a-82ce-f87e0ae086c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7277fd6-62shf\" (UID: \"1ea379a7-6815-4d5a-82ce-f87e0ae086c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7277fd6-62shf" Oct 30 21:39:55.828114 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:55.828099 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ea379a7-6815-4d5a-82ce-f87e0ae086c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7277fd6-62shf\" (UID: \"1ea379a7-6815-4d5a-82ce-f87e0ae086c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7277fd6-62shf" Oct 30 21:39:55.866537 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:55.866499 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:39:55.866537 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:55.866500 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:39:55.866537 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:55.866504 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:39:55.868988 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:55.868962 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7277fd6-62shf"] Oct 30 21:39:55.915803 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:55.915776 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 21:39:55.928993 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:55.928966 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8v8k\" (UniqueName: \"kubernetes.io/projected/1ea379a7-6815-4d5a-82ce-f87e0ae086c3-kube-api-access-x8v8k\") pod \"maas-api-revoke-sub-b7277fd6-62shf\" (UID: \"1ea379a7-6815-4d5a-82ce-f87e0ae086c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7277fd6-62shf" Oct 30 21:39:55.929111 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:55.929014 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ea379a7-6815-4d5a-82ce-f87e0ae086c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7277fd6-62shf\" (UID: \"1ea379a7-6815-4d5a-82ce-f87e0ae086c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7277fd6-62shf" Oct 30 21:39:55.929111 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:55.929076 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ea379a7-6815-4d5a-82ce-f87e0ae086c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7277fd6-62shf\" (UID: \"1ea379a7-6815-4d5a-82ce-f87e0ae086c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7277fd6-62shf" Oct 30 21:39:55.929658 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:55.929638 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ea379a7-6815-4d5a-82ce-f87e0ae086c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7277fd6-62shf\" (UID: \"1ea379a7-6815-4d5a-82ce-f87e0ae086c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7277fd6-62shf" Oct 30 21:39:55.931538 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:55.931510 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ea379a7-6815-4d5a-82ce-f87e0ae086c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7277fd6-62shf\" (UID: \"1ea379a7-6815-4d5a-82ce-f87e0ae086c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7277fd6-62shf" Oct 30 21:39:55.990552 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:55.990515 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8v8k\" (UniqueName: \"kubernetes.io/projected/1ea379a7-6815-4d5a-82ce-f87e0ae086c3-kube-api-access-x8v8k\") pod \"maas-api-revoke-sub-b7277fd6-62shf\" (UID: \"1ea379a7-6815-4d5a-82ce-f87e0ae086c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7277fd6-62shf" Oct 30 21:39:56.128125 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:56.128038 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7277fd6-62shf" Oct 30 21:39:56.511449 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:56.511413 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7277fd6-62shf"] Oct 30 21:39:56.512391 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:39:56.512361 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ea379a7_6815_4d5a_82ce_f87e0ae086c3.slice/crio-3acd3e8bdcabf06a869fa8732edf58bc3daddf41eb7eddc79b9fd4b241654a63 WatchSource:0}: Error finding container 3acd3e8bdcabf06a869fa8732edf58bc3daddf41eb7eddc79b9fd4b241654a63: Status 404 returned error can't find the container with id 3acd3e8bdcabf06a869fa8732edf58bc3daddf41eb7eddc79b9fd4b241654a63 Oct 30 21:39:57.318092 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:57.318056 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7277fd6-62shf" event={"ID":"1ea379a7-6815-4d5a-82ce-f87e0ae086c3","Type":"ContainerStarted","Data":"ef6132ff756c537ef6be6af76156e32979c1b73dfe271127d485ddcfc7e64662"} Oct 30 21:39:57.318474 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:57.318111 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7277fd6-62shf" event={"ID":"1ea379a7-6815-4d5a-82ce-f87e0ae086c3","Type":"ContainerStarted","Data":"3acd3e8bdcabf06a869fa8732edf58bc3daddf41eb7eddc79b9fd4b241654a63"} Oct 30 21:39:57.437746 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:57.437687 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7277fd6-62shf" podStartSLOduration=2.437673438 podStartE2EDuration="2.437673438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:57.434472549 +0000 UTC m=+981.176993475" watchObservedRunningTime="2026-10-30 21:39:57.437673438 +0000 UTC m=+981.180194364" Oct 30 21:39:58.221912 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:58.221869 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2977c254-smh4c"] Oct 30 21:39:58.225837 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:58.225812 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2977c254-smh4c" Oct 30 21:39:58.251917 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:58.251874 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae34ed3b-a844-4f73-bb27-4c75ba279377-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2977c254-smh4c\" (UID: \"ae34ed3b-a844-4f73-bb27-4c75ba279377\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2977c254-smh4c" Oct 30 21:39:58.252064 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:58.251948 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae34ed3b-a844-4f73-bb27-4c75ba279377-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2977c254-smh4c\" (UID: \"ae34ed3b-a844-4f73-bb27-4c75ba279377\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2977c254-smh4c" Oct 30 21:39:58.252216 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:58.252194 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2f9d\" (UniqueName: \"kubernetes.io/projected/ae34ed3b-a844-4f73-bb27-4c75ba279377-kube-api-access-x2f9d\") pod \"maas-api-revoke-sub-2977c254-smh4c\" (UID: \"ae34ed3b-a844-4f73-bb27-4c75ba279377\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2977c254-smh4c" Oct 30 21:39:58.285039 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:58.285008 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2977c254-smh4c"] Oct 30 21:39:58.326664 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:58.326636 5748 generic.go:356] "Generic (PLEG): container finished" podID="1ea379a7-6815-4d5a-82ce-f87e0ae086c3" containerID="ef6132ff756c537ef6be6af76156e32979c1b73dfe271127d485ddcfc7e64662" exitCode=0 Oct 30 21:39:58.327060 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:58.326744 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7277fd6-62shf" event={"ID":"1ea379a7-6815-4d5a-82ce-f87e0ae086c3","Type":"ContainerDied","Data":"ef6132ff756c537ef6be6af76156e32979c1b73dfe271127d485ddcfc7e64662"} Oct 30 21:39:58.352933 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:58.352880 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae34ed3b-a844-4f73-bb27-4c75ba279377-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2977c254-smh4c\" (UID: \"ae34ed3b-a844-4f73-bb27-4c75ba279377\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2977c254-smh4c" Oct 30 21:39:58.353092 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:58.352942 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae34ed3b-a844-4f73-bb27-4c75ba279377-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2977c254-smh4c\" (UID: \"ae34ed3b-a844-4f73-bb27-4c75ba279377\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2977c254-smh4c" Oct 30 21:39:58.353092 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:58.353084 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2f9d\" (UniqueName: \"kubernetes.io/projected/ae34ed3b-a844-4f73-bb27-4c75ba279377-kube-api-access-x2f9d\") pod \"maas-api-revoke-sub-2977c254-smh4c\" (UID: \"ae34ed3b-a844-4f73-bb27-4c75ba279377\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2977c254-smh4c" Oct 30 21:39:58.353639 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:58.353613 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae34ed3b-a844-4f73-bb27-4c75ba279377-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2977c254-smh4c\" (UID: \"ae34ed3b-a844-4f73-bb27-4c75ba279377\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2977c254-smh4c" Oct 30 21:39:58.355227 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:58.355207 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae34ed3b-a844-4f73-bb27-4c75ba279377-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2977c254-smh4c\" (UID: \"ae34ed3b-a844-4f73-bb27-4c75ba279377\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2977c254-smh4c" Oct 30 21:39:58.414849 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:58.414824 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2f9d\" (UniqueName: \"kubernetes.io/projected/ae34ed3b-a844-4f73-bb27-4c75ba279377-kube-api-access-x2f9d\") pod \"maas-api-revoke-sub-2977c254-smh4c\" (UID: \"ae34ed3b-a844-4f73-bb27-4c75ba279377\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2977c254-smh4c" Oct 30 21:39:58.541461 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:58.541366 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2977c254-smh4c" Oct 30 21:39:58.918089 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:58.918058 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2977c254-smh4c"] Oct 30 21:39:58.919772 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:39:58.919749 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae34ed3b_a844_4f73_bb27_4c75ba279377.slice/crio-0b74ce860a876d6165e5e2003f87f70f3f36ad171364be524e8d50efc92cc301 WatchSource:0}: Error finding container 0b74ce860a876d6165e5e2003f87f70f3f36ad171364be524e8d50efc92cc301: Status 404 returned error can't find the container with id 0b74ce860a876d6165e5e2003f87f70f3f36ad171364be524e8d50efc92cc301 Oct 30 21:39:59.334026 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:59.333927 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2977c254-smh4c" event={"ID":"ae34ed3b-a844-4f73-bb27-4c75ba279377","Type":"ContainerStarted","Data":"278c01d11609244128c36df28a970d20fe331901701f8b45a151a73141d8cdbf"} Oct 30 21:39:59.334026 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:59.333984 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2977c254-smh4c" event={"ID":"ae34ed3b-a844-4f73-bb27-4c75ba279377","Type":"ContainerStarted","Data":"0b74ce860a876d6165e5e2003f87f70f3f36ad171364be524e8d50efc92cc301"} Oct 30 21:39:59.432919 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:59.428576 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7"] Oct 30 21:39:59.434117 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:59.434087 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7" Oct 30 21:39:59.463687 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:59.463660 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2cdb6d5-0628-435d-ade1-e0aa84440718-workload-certs\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7\" (UID: \"d2cdb6d5-0628-435d-ade1-e0aa84440718\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7" Oct 30 21:39:59.463916 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:59.463714 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2cdb6d5-0628-435d-ade1-e0aa84440718-istio-podinfo\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7\" (UID: \"d2cdb6d5-0628-435d-ade1-e0aa84440718\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7" Oct 30 21:39:59.463916 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:59.463844 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2cdb6d5-0628-435d-ade1-e0aa84440718-istio-envoy\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7\" (UID: \"d2cdb6d5-0628-435d-ade1-e0aa84440718\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7" Oct 30 21:39:59.464058 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:59.463891 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2cdb6d5-0628-435d-ade1-e0aa84440718-istiod-ca-cert\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7\" (UID: \"d2cdb6d5-0628-435d-ade1-e0aa84440718\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7" Oct 30 21:39:59.464117 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:59.464075 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2cdb6d5-0628-435d-ade1-e0aa84440718-istio-data\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7\" (UID: \"d2cdb6d5-0628-435d-ade1-e0aa84440718\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7" Oct 30 21:39:59.464189 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:59.464168 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c87t2\" (UniqueName: \"kubernetes.io/projected/d2cdb6d5-0628-435d-ade1-e0aa84440718-kube-api-access-c87t2\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7\" (UID: \"d2cdb6d5-0628-435d-ade1-e0aa84440718\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7" Oct 30 21:39:59.464262 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:59.464244 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2cdb6d5-0628-435d-ade1-e0aa84440718-workload-socket\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7\" (UID: \"d2cdb6d5-0628-435d-ade1-e0aa84440718\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7" Oct 30 21:39:59.464384 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:59.464366 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2cdb6d5-0628-435d-ade1-e0aa84440718-credential-socket\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7\" (UID: \"d2cdb6d5-0628-435d-ade1-e0aa84440718\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7" Oct 30 21:39:59.464464 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:59.464404 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2cdb6d5-0628-435d-ade1-e0aa84440718-istio-token\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7\" (UID: \"d2cdb6d5-0628-435d-ade1-e0aa84440718\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7" Oct 30 21:39:59.490404 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:59.490360 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-91874e-openshift-default-dockercfg-zjpd2\"" Oct 30 21:39:59.493656 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:59.493606 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2977c254-smh4c" podStartSLOduration=1.4935925509999999 podStartE2EDuration="1.493592551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:59.458611881 +0000 UTC m=+983.201132816" watchObservedRunningTime="2026-10-30 21:39:59.493592551 +0000 UTC m=+983.236113485" Oct 30 21:39:59.493805 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:59.493765 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7"] Oct 30 21:39:59.504833 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:59.504806 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7277fd6-62shf" Oct 30 21:39:59.565240 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:59.565207 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ea379a7-6815-4d5a-82ce-f87e0ae086c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ea379a7-6815-4d5a-82ce-f87e0ae086c3-maas-api-xks-ca\") pod \"1ea379a7-6815-4d5a-82ce-f87e0ae086c3\" (UID: \"1ea379a7-6815-4d5a-82ce-f87e0ae086c3\") " Oct 30 21:39:59.565411 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:59.565359 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ea379a7-6815-4d5a-82ce-f87e0ae086c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ea379a7-6815-4d5a-82ce-f87e0ae086c3-maas-api-openshift-ca\") pod \"1ea379a7-6815-4d5a-82ce-f87e0ae086c3\" (UID: \"1ea379a7-6815-4d5a-82ce-f87e0ae086c3\") " Oct 30 21:39:59.565461 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:59.565417 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ea379a7-6815-4d5a-82ce-f87e0ae086c3-kube-api-access-x8v8k\" (UniqueName: \"kubernetes.io/projected/1ea379a7-6815-4d5a-82ce-f87e0ae086c3-kube-api-access-x8v8k\") pod \"1ea379a7-6815-4d5a-82ce-f87e0ae086c3\" (UID: \"1ea379a7-6815-4d5a-82ce-f87e0ae086c3\") " Oct 30 21:39:59.565579 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:59.565562 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2cdb6d5-0628-435d-ade1-e0aa84440718-istio-token\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7\" (UID: \"d2cdb6d5-0628-435d-ade1-e0aa84440718\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7" Oct 30 21:39:59.565656 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:59.565641 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2cdb6d5-0628-435d-ade1-e0aa84440718-workload-certs\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7\" (UID: \"d2cdb6d5-0628-435d-ade1-e0aa84440718\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7" Oct 30 21:39:59.565730 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:59.565699 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2cdb6d5-0628-435d-ade1-e0aa84440718-istio-podinfo\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7\" (UID: \"d2cdb6d5-0628-435d-ade1-e0aa84440718\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7" Oct 30 21:39:59.565788 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:59.565747 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ea379a7-6815-4d5a-82ce-f87e0ae086c3-maas-api-openshift-ca" pod "1ea379a7-6815-4d5a-82ce-f87e0ae086c3" (UID: "1ea379a7-6815-4d5a-82ce-f87e0ae086c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:39:59.565788 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:59.565760 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2cdb6d5-0628-435d-ade1-e0aa84440718-istio-envoy\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7\" (UID: \"d2cdb6d5-0628-435d-ade1-e0aa84440718\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7" Oct 30 21:39:59.566232 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:59.565830 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2cdb6d5-0628-435d-ade1-e0aa84440718-istiod-ca-cert\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7\" (UID: \"d2cdb6d5-0628-435d-ade1-e0aa84440718\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7" Oct 30 21:39:59.566232 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:59.565884 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2cdb6d5-0628-435d-ade1-e0aa84440718-istio-data\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7\" (UID: \"d2cdb6d5-0628-435d-ade1-e0aa84440718\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7" Oct 30 21:39:59.566232 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:59.565958 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c87t2\" (UniqueName: \"kubernetes.io/projected/d2cdb6d5-0628-435d-ade1-e0aa84440718-kube-api-access-c87t2\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7\" (UID: \"d2cdb6d5-0628-435d-ade1-e0aa84440718\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7" Oct 30 21:39:59.566232 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:59.566089 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2cdb6d5-0628-435d-ade1-e0aa84440718-workload-socket\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7\" (UID: \"d2cdb6d5-0628-435d-ade1-e0aa84440718\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7" Oct 30 21:39:59.566232 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:59.566114 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2cdb6d5-0628-435d-ade1-e0aa84440718-credential-socket\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7\" (UID: \"d2cdb6d5-0628-435d-ade1-e0aa84440718\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7" Oct 30 21:39:59.566232 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:59.566186 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ea379a7-6815-4d5a-82ce-f87e0ae086c3-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:59.567759 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:59.566234 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2cdb6d5-0628-435d-ade1-e0aa84440718-workload-certs\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7\" (UID: \"d2cdb6d5-0628-435d-ade1-e0aa84440718\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7" Oct 30 21:39:59.567759 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:59.566453 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2cdb6d5-0628-435d-ade1-e0aa84440718-credential-socket\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7\" (UID: \"d2cdb6d5-0628-435d-ade1-e0aa84440718\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7" Oct 30 21:39:59.567759 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:59.566700 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2cdb6d5-0628-435d-ade1-e0aa84440718-istio-data\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7\" (UID: \"d2cdb6d5-0628-435d-ade1-e0aa84440718\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7" Oct 30 21:39:59.567759 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:59.566743 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2cdb6d5-0628-435d-ade1-e0aa84440718-workload-socket\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7\" (UID: \"d2cdb6d5-0628-435d-ade1-e0aa84440718\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7" Oct 30 21:39:59.568265 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:59.568237 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2cdb6d5-0628-435d-ade1-e0aa84440718-istiod-ca-cert\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7\" (UID: \"d2cdb6d5-0628-435d-ade1-e0aa84440718\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7" Oct 30 21:39:59.568488 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:59.568466 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ea379a7-6815-4d5a-82ce-f87e0ae086c3-maas-api-xks-ca" pod "1ea379a7-6815-4d5a-82ce-f87e0ae086c3" (UID: "1ea379a7-6815-4d5a-82ce-f87e0ae086c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:39:59.568629 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:59.568605 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ea379a7-6815-4d5a-82ce-f87e0ae086c3-kube-api-access-x8v8k" pod "1ea379a7-6815-4d5a-82ce-f87e0ae086c3" (UID: "1ea379a7-6815-4d5a-82ce-f87e0ae086c3"). InnerVolumeSpecName "kube-api-access-x8v8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:39:59.569088 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:59.569054 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2cdb6d5-0628-435d-ade1-e0aa84440718-istio-podinfo\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7\" (UID: \"d2cdb6d5-0628-435d-ade1-e0aa84440718\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7" Oct 30 21:39:59.569368 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:59.569348 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2cdb6d5-0628-435d-ade1-e0aa84440718-istio-envoy\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7\" (UID: \"d2cdb6d5-0628-435d-ade1-e0aa84440718\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7" Oct 30 21:39:59.627599 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:59.627536 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c87t2\" (UniqueName: \"kubernetes.io/projected/d2cdb6d5-0628-435d-ade1-e0aa84440718-kube-api-access-c87t2\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7\" (UID: \"d2cdb6d5-0628-435d-ade1-e0aa84440718\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7" Oct 30 21:39:59.628887 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:59.628866 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2cdb6d5-0628-435d-ade1-e0aa84440718-istio-token\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7\" (UID: \"d2cdb6d5-0628-435d-ade1-e0aa84440718\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7" Oct 30 21:39:59.667231 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:59.667197 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ea379a7-6815-4d5a-82ce-f87e0ae086c3-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:59.667231 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:59.667233 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8v8k\" (UniqueName: \"kubernetes.io/projected/1ea379a7-6815-4d5a-82ce-f87e0ae086c3-kube-api-access-x8v8k\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:59.766311 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:39:59.766278 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7" Oct 30 21:40:00.145312 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:40:00.145275 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:40:00.202785 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:00.202753 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7"] Oct 30 21:40:00.204264 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:40:00.204238 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2cdb6d5_0628_435d_ade1_e0aa84440718.slice/crio-e36ab17ca70ef7ee51e8967050226d15bd689b65675e10e6a101ea913dde862b WatchSource:0}: Error finding container e36ab17ca70ef7ee51e8967050226d15bd689b65675e10e6a101ea913dde862b: Status 404 returned error can't find the container with id e36ab17ca70ef7ee51e8967050226d15bd689b65675e10e6a101ea913dde862b Oct 30 21:40:00.206716 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:00.206685 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:40:00.206825 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:00.206761 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:40:00.206825 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:00.206813 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:40:00.341601 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:00.341563 5748 generic.go:356] "Generic (PLEG): container finished" podID="ae34ed3b-a844-4f73-bb27-4c75ba279377" containerID="278c01d11609244128c36df28a970d20fe331901701f8b45a151a73141d8cdbf" exitCode=0 Oct 30 21:40:00.342159 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:00.341654 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2977c254-smh4c" event={"ID":"ae34ed3b-a844-4f73-bb27-4c75ba279377","Type":"ContainerDied","Data":"278c01d11609244128c36df28a970d20fe331901701f8b45a151a73141d8cdbf"} Oct 30 21:40:00.343343 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:00.343313 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7277fd6-62shf" Oct 30 21:40:00.343491 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:00.343362 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7277fd6-62shf" event={"ID":"1ea379a7-6815-4d5a-82ce-f87e0ae086c3","Type":"ContainerDied","Data":"3acd3e8bdcabf06a869fa8732edf58bc3daddf41eb7eddc79b9fd4b241654a63"} Oct 30 21:40:00.343491 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:00.343417 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3acd3e8bdcabf06a869fa8732edf58bc3daddf41eb7eddc79b9fd4b241654a63" Oct 30 21:40:00.345285 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:00.345256 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7" event={"ID":"d2cdb6d5-0628-435d-ade1-e0aa84440718","Type":"ContainerStarted","Data":"2caff4996f2261f9c21a7bb1a193e9df5906df30f09c82b14ecfef46f90dcd86"} Oct 30 21:40:00.345410 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:00.345306 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7" event={"ID":"d2cdb6d5-0628-435d-ade1-e0aa84440718","Type":"ContainerStarted","Data":"e36ab17ca70ef7ee51e8967050226d15bd689b65675e10e6a101ea913dde862b"} Oct 30 21:40:00.509857 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:00.509757 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b6b39cc-kjgl2"] Oct 30 21:40:00.510335 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:00.510316 5748 cpu_manager.go:414] "containerMap: removing container" podUID="1ea379a7-6815-4d5a-82ce-f87e0ae086c3" containerName="revoke-keys" Oct 30 21:40:00.510335 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:00.510333 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ea379a7-6815-4d5a-82ce-f87e0ae086c3" containerName="revoke-keys" Oct 30 21:40:00.510535 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:00.510449 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ea379a7-6815-4d5a-82ce-f87e0ae086c3" containerName="revoke-keys" Oct 30 21:40:00.513967 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:00.513945 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6b39cc-kjgl2" Oct 30 21:40:00.572002 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:00.571967 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b6b39cc-kjgl2"] Oct 30 21:40:00.576626 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:00.576596 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f3aa9b8-02e6-4e1d-856d-7e6507d6c220-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b6b39cc-kjgl2\" (UID: \"5f3aa9b8-02e6-4e1d-856d-7e6507d6c220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6b39cc-kjgl2" Oct 30 21:40:00.576767 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:00.576739 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f3aa9b8-02e6-4e1d-856d-7e6507d6c220-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b6b39cc-kjgl2\" (UID: \"5f3aa9b8-02e6-4e1d-856d-7e6507d6c220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6b39cc-kjgl2" Oct 30 21:40:00.576843 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:00.576784 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-529jk\" (UniqueName: \"kubernetes.io/projected/5f3aa9b8-02e6-4e1d-856d-7e6507d6c220-kube-api-access-529jk\") pod \"maas-api-revoke-sub-1b6b39cc-kjgl2\" (UID: \"5f3aa9b8-02e6-4e1d-856d-7e6507d6c220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6b39cc-kjgl2" Oct 30 21:40:00.634510 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:00.634444 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7" podStartSLOduration=1.634426862 podStartE2EDuration="1.634426862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:00.633420126 +0000 UTC m=+984.375941062" watchObservedRunningTime="2026-10-30 21:40:00.634426862 +0000 UTC m=+984.376947801" Oct 30 21:40:00.678804 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:00.677812 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f3aa9b8-02e6-4e1d-856d-7e6507d6c220-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b6b39cc-kjgl2\" (UID: \"5f3aa9b8-02e6-4e1d-856d-7e6507d6c220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6b39cc-kjgl2" Oct 30 21:40:00.678804 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:00.678515 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f3aa9b8-02e6-4e1d-856d-7e6507d6c220-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b6b39cc-kjgl2\" (UID: \"5f3aa9b8-02e6-4e1d-856d-7e6507d6c220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6b39cc-kjgl2" Oct 30 21:40:00.678804 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:00.678599 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-529jk\" (UniqueName: \"kubernetes.io/projected/5f3aa9b8-02e6-4e1d-856d-7e6507d6c220-kube-api-access-529jk\") pod \"maas-api-revoke-sub-1b6b39cc-kjgl2\" (UID: \"5f3aa9b8-02e6-4e1d-856d-7e6507d6c220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6b39cc-kjgl2" Oct 30 21:40:00.679718 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:00.679642 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f3aa9b8-02e6-4e1d-856d-7e6507d6c220-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b6b39cc-kjgl2\" (UID: \"5f3aa9b8-02e6-4e1d-856d-7e6507d6c220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6b39cc-kjgl2" Oct 30 21:40:00.686920 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:00.682503 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f3aa9b8-02e6-4e1d-856d-7e6507d6c220-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b6b39cc-kjgl2\" (UID: \"5f3aa9b8-02e6-4e1d-856d-7e6507d6c220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6b39cc-kjgl2" Oct 30 21:40:00.745984 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:00.745954 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-529jk\" (UniqueName: \"kubernetes.io/projected/5f3aa9b8-02e6-4e1d-856d-7e6507d6c220-kube-api-access-529jk\") pod \"maas-api-revoke-sub-1b6b39cc-kjgl2\" (UID: \"5f3aa9b8-02e6-4e1d-856d-7e6507d6c220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6b39cc-kjgl2" Oct 30 21:40:00.766454 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:00.766387 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7" Oct 30 21:40:00.831071 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:00.831028 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6b39cc-kjgl2" Oct 30 21:40:01.229088 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:40:01.229053 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f3aa9b8_02e6_4e1d_856d_7e6507d6c220.slice/crio-03c1a531f20c7be440ce29523c0954337c9ce9a4c9719c465a9ce55de2385178 WatchSource:0}: Error finding container 03c1a531f20c7be440ce29523c0954337c9ce9a4c9719c465a9ce55de2385178: Status 404 returned error can't find the container with id 03c1a531f20c7be440ce29523c0954337c9ce9a4c9719c465a9ce55de2385178 Oct 30 21:40:01.229659 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:01.229639 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b6b39cc-kjgl2"] Oct 30 21:40:01.352935 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:01.352884 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6b39cc-kjgl2" event={"ID":"5f3aa9b8-02e6-4e1d-856d-7e6507d6c220","Type":"ContainerStarted","Data":"03c1a531f20c7be440ce29523c0954337c9ce9a4c9719c465a9ce55de2385178"} Oct 30 21:40:01.592514 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:01.592484 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2977c254-smh4c" Oct 30 21:40:01.691399 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:01.691360 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae34ed3b-a844-4f73-bb27-4c75ba279377-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae34ed3b-a844-4f73-bb27-4c75ba279377-maas-api-openshift-ca\") pod \"ae34ed3b-a844-4f73-bb27-4c75ba279377\" (UID: \"ae34ed3b-a844-4f73-bb27-4c75ba279377\") " Oct 30 21:40:01.691580 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:01.691523 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae34ed3b-a844-4f73-bb27-4c75ba279377-kube-api-access-x2f9d\" (UniqueName: \"kubernetes.io/projected/ae34ed3b-a844-4f73-bb27-4c75ba279377-kube-api-access-x2f9d\") pod \"ae34ed3b-a844-4f73-bb27-4c75ba279377\" (UID: \"ae34ed3b-a844-4f73-bb27-4c75ba279377\") " Oct 30 21:40:01.691633 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:01.691605 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae34ed3b-a844-4f73-bb27-4c75ba279377-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae34ed3b-a844-4f73-bb27-4c75ba279377-maas-api-xks-ca\") pod \"ae34ed3b-a844-4f73-bb27-4c75ba279377\" (UID: \"ae34ed3b-a844-4f73-bb27-4c75ba279377\") " Oct 30 21:40:01.691800 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:01.691775 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae34ed3b-a844-4f73-bb27-4c75ba279377-maas-api-openshift-ca" pod "ae34ed3b-a844-4f73-bb27-4c75ba279377" (UID: "ae34ed3b-a844-4f73-bb27-4c75ba279377"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:40:01.692043 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:01.692024 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae34ed3b-a844-4f73-bb27-4c75ba279377-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:40:01.693641 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:01.693618 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae34ed3b-a844-4f73-bb27-4c75ba279377-maas-api-xks-ca" pod "ae34ed3b-a844-4f73-bb27-4c75ba279377" (UID: "ae34ed3b-a844-4f73-bb27-4c75ba279377"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:40:01.693876 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:01.693854 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae34ed3b-a844-4f73-bb27-4c75ba279377-kube-api-access-x2f9d" pod "ae34ed3b-a844-4f73-bb27-4c75ba279377" (UID: "ae34ed3b-a844-4f73-bb27-4c75ba279377"). InnerVolumeSpecName "kube-api-access-x2f9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:40:01.767879 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:01.767798 5748 patch_prober.go:29] interesting pod/e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 21:40:01.767879 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:01.767855 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7" podUID="d2cdb6d5-0628-435d-ade1-e0aa84440718" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 21:40:01.793621 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:01.793593 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2f9d\" (UniqueName: \"kubernetes.io/projected/ae34ed3b-a844-4f73-bb27-4c75ba279377-kube-api-access-x2f9d\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:40:01.793621 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:01.793620 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae34ed3b-a844-4f73-bb27-4c75ba279377-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:40:02.361335 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:02.361305 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2977c254-smh4c" Oct 30 21:40:02.361768 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:02.361340 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2977c254-smh4c" event={"ID":"ae34ed3b-a844-4f73-bb27-4c75ba279377","Type":"ContainerDied","Data":"0b74ce860a876d6165e5e2003f87f70f3f36ad171364be524e8d50efc92cc301"} Oct 30 21:40:02.361768 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:02.361396 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b74ce860a876d6165e5e2003f87f70f3f36ad171364be524e8d50efc92cc301" Oct 30 21:40:02.363203 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:02.363176 5748 generic.go:356] "Generic (PLEG): container finished" podID="5f3aa9b8-02e6-4e1d-856d-7e6507d6c220" containerID="4d9ab051c9273ee2a695b9da12a63e55226755c79e4e88b4949dd5e6af5f7b3a" exitCode=0 Oct 30 21:40:02.363314 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:02.363259 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6b39cc-kjgl2" event={"ID":"5f3aa9b8-02e6-4e1d-856d-7e6507d6c220","Type":"ContainerDied","Data":"4d9ab051c9273ee2a695b9da12a63e55226755c79e4e88b4949dd5e6af5f7b3a"} Oct 30 21:40:02.767658 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:02.767612 5748 patch_prober.go:29] interesting pod/e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 21:40:02.767820 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:02.767672 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7" podUID="d2cdb6d5-0628-435d-ade1-e0aa84440718" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 21:40:03.560101 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:03.560074 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6b39cc-kjgl2" Oct 30 21:40:03.636658 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:03.636579 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f3aa9b8-02e6-4e1d-856d-7e6507d6c220-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f3aa9b8-02e6-4e1d-856d-7e6507d6c220-maas-api-openshift-ca\") pod \"5f3aa9b8-02e6-4e1d-856d-7e6507d6c220\" (UID: \"5f3aa9b8-02e6-4e1d-856d-7e6507d6c220\") " Oct 30 21:40:03.636658 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:03.636642 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f3aa9b8-02e6-4e1d-856d-7e6507d6c220-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f3aa9b8-02e6-4e1d-856d-7e6507d6c220-maas-api-xks-ca\") pod \"5f3aa9b8-02e6-4e1d-856d-7e6507d6c220\" (UID: \"5f3aa9b8-02e6-4e1d-856d-7e6507d6c220\") " Oct 30 21:40:03.636820 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:03.636732 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f3aa9b8-02e6-4e1d-856d-7e6507d6c220-kube-api-access-529jk\" (UniqueName: \"kubernetes.io/projected/5f3aa9b8-02e6-4e1d-856d-7e6507d6c220-kube-api-access-529jk\") pod \"5f3aa9b8-02e6-4e1d-856d-7e6507d6c220\" (UID: \"5f3aa9b8-02e6-4e1d-856d-7e6507d6c220\") " Oct 30 21:40:03.636947 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:03.636922 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f3aa9b8-02e6-4e1d-856d-7e6507d6c220-maas-api-openshift-ca" pod "5f3aa9b8-02e6-4e1d-856d-7e6507d6c220" (UID: "5f3aa9b8-02e6-4e1d-856d-7e6507d6c220"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:40:03.637161 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:03.637138 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f3aa9b8-02e6-4e1d-856d-7e6507d6c220-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:40:03.638707 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:03.638681 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f3aa9b8-02e6-4e1d-856d-7e6507d6c220-maas-api-xks-ca" pod "5f3aa9b8-02e6-4e1d-856d-7e6507d6c220" (UID: "5f3aa9b8-02e6-4e1d-856d-7e6507d6c220"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:40:03.638802 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:03.638719 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f3aa9b8-02e6-4e1d-856d-7e6507d6c220-kube-api-access-529jk" pod "5f3aa9b8-02e6-4e1d-856d-7e6507d6c220" (UID: "5f3aa9b8-02e6-4e1d-856d-7e6507d6c220"). InnerVolumeSpecName "kube-api-access-529jk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:40:03.738522 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:03.738480 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f3aa9b8-02e6-4e1d-856d-7e6507d6c220-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:40:03.738675 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:03.738564 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-529jk\" (UniqueName: \"kubernetes.io/projected/5f3aa9b8-02e6-4e1d-856d-7e6507d6c220-kube-api-access-529jk\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:40:03.767421 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:03.767384 5748 patch_prober.go:29] interesting pod/e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 21:40:03.767600 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:03.767444 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7" podUID="d2cdb6d5-0628-435d-ade1-e0aa84440718" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 21:40:04.375662 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:04.375632 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6b39cc-kjgl2" Oct 30 21:40:04.375820 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:04.375669 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6b39cc-kjgl2" event={"ID":"5f3aa9b8-02e6-4e1d-856d-7e6507d6c220","Type":"ContainerDied","Data":"03c1a531f20c7be440ce29523c0954337c9ce9a4c9719c465a9ce55de2385178"} Oct 30 21:40:04.375820 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:04.375715 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03c1a531f20c7be440ce29523c0954337c9ce9a4c9719c465a9ce55de2385178" Oct 30 21:40:04.766650 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:04.766611 5748 patch_prober.go:29] interesting pod/e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 21:40:04.767055 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:04.766671 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7" podUID="d2cdb6d5-0628-435d-ade1-e0aa84440718" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 21:40:05.686777 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:05.686739 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-khvpj"] Oct 30 21:40:05.687250 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:05.687236 5748 cpu_manager.go:414] "containerMap: removing container" podUID="5f3aa9b8-02e6-4e1d-856d-7e6507d6c220" containerName="revoke-keys" Oct 30 21:40:05.687299 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:05.687251 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f3aa9b8-02e6-4e1d-856d-7e6507d6c220" containerName="revoke-keys" Oct 30 21:40:05.687299 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:05.687274 5748 cpu_manager.go:414] "containerMap: removing container" podUID="ae34ed3b-a844-4f73-bb27-4c75ba279377" containerName="revoke-keys" Oct 30 21:40:05.687299 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:05.687280 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae34ed3b-a844-4f73-bb27-4c75ba279377" containerName="revoke-keys" Oct 30 21:40:05.687397 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:05.687388 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae34ed3b-a844-4f73-bb27-4c75ba279377" containerName="revoke-keys" Oct 30 21:40:05.687432 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:05.687404 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f3aa9b8-02e6-4e1d-856d-7e6507d6c220" containerName="revoke-keys" Oct 30 21:40:05.691078 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:05.691057 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-khvpj" Oct 30 21:40:05.744074 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:05.744036 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-c7d477-openshift-default-dockercfg-q6mxs\"" Oct 30 21:40:05.758084 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:05.758048 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39e20f10-f92b-4919-b80d-519ad1120294-credential-socket\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-khvpj\" (UID: \"39e20f10-f92b-4919-b80d-519ad1120294\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-khvpj" Oct 30 21:40:05.758084 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:05.758088 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39e20f10-f92b-4919-b80d-519ad1120294-istio-token\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-khvpj\" (UID: \"39e20f10-f92b-4919-b80d-519ad1120294\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-khvpj" Oct 30 21:40:05.758339 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:05.758106 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39e20f10-f92b-4919-b80d-519ad1120294-istiod-ca-cert\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-khvpj\" (UID: \"39e20f10-f92b-4919-b80d-519ad1120294\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-khvpj" Oct 30 21:40:05.758339 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:05.758151 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39e20f10-f92b-4919-b80d-519ad1120294-workload-socket\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-khvpj\" (UID: \"39e20f10-f92b-4919-b80d-519ad1120294\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-khvpj" Oct 30 21:40:05.758339 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:05.758224 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jht5g\" (UniqueName: \"kubernetes.io/projected/39e20f10-f92b-4919-b80d-519ad1120294-kube-api-access-jht5g\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-khvpj\" (UID: \"39e20f10-f92b-4919-b80d-519ad1120294\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-khvpj" Oct 30 21:40:05.758339 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:05.758274 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39e20f10-f92b-4919-b80d-519ad1120294-istio-envoy\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-khvpj\" (UID: \"39e20f10-f92b-4919-b80d-519ad1120294\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-khvpj" Oct 30 21:40:05.758339 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:05.758297 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39e20f10-f92b-4919-b80d-519ad1120294-istio-data\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-khvpj\" (UID: \"39e20f10-f92b-4919-b80d-519ad1120294\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-khvpj" Oct 30 21:40:05.758339 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:05.758320 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39e20f10-f92b-4919-b80d-519ad1120294-workload-certs\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-khvpj\" (UID: \"39e20f10-f92b-4919-b80d-519ad1120294\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-khvpj" Oct 30 21:40:05.758723 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:05.758348 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39e20f10-f92b-4919-b80d-519ad1120294-istio-podinfo\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-khvpj\" (UID: \"39e20f10-f92b-4919-b80d-519ad1120294\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-khvpj" Oct 30 21:40:05.767210 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:05.767165 5748 patch_prober.go:29] interesting pod/e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 21:40:05.767658 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:05.767226 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7" podUID="d2cdb6d5-0628-435d-ade1-e0aa84440718" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 21:40:05.797520 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:05.797483 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-khvpj"] Oct 30 21:40:05.859224 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:05.859188 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39e20f10-f92b-4919-b80d-519ad1120294-istio-podinfo\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-khvpj\" (UID: \"39e20f10-f92b-4919-b80d-519ad1120294\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-khvpj" Oct 30 21:40:05.859421 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:05.859264 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39e20f10-f92b-4919-b80d-519ad1120294-credential-socket\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-khvpj\" (UID: \"39e20f10-f92b-4919-b80d-519ad1120294\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-khvpj" Oct 30 21:40:05.859421 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:05.859281 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39e20f10-f92b-4919-b80d-519ad1120294-istio-token\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-khvpj\" (UID: \"39e20f10-f92b-4919-b80d-519ad1120294\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-khvpj" Oct 30 21:40:05.859421 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:05.859297 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39e20f10-f92b-4919-b80d-519ad1120294-istiod-ca-cert\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-khvpj\" (UID: \"39e20f10-f92b-4919-b80d-519ad1120294\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-khvpj" Oct 30 21:40:05.859421 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:05.859316 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39e20f10-f92b-4919-b80d-519ad1120294-workload-socket\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-khvpj\" (UID: \"39e20f10-f92b-4919-b80d-519ad1120294\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-khvpj" Oct 30 21:40:05.859421 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:05.859367 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jht5g\" (UniqueName: \"kubernetes.io/projected/39e20f10-f92b-4919-b80d-519ad1120294-kube-api-access-jht5g\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-khvpj\" (UID: \"39e20f10-f92b-4919-b80d-519ad1120294\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-khvpj" Oct 30 21:40:05.859421 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:05.859412 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39e20f10-f92b-4919-b80d-519ad1120294-istio-envoy\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-khvpj\" (UID: \"39e20f10-f92b-4919-b80d-519ad1120294\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-khvpj" Oct 30 21:40:05.859737 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:05.859434 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39e20f10-f92b-4919-b80d-519ad1120294-istio-data\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-khvpj\" (UID: \"39e20f10-f92b-4919-b80d-519ad1120294\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-khvpj" Oct 30 21:40:05.859737 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:05.859451 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39e20f10-f92b-4919-b80d-519ad1120294-workload-certs\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-khvpj\" (UID: \"39e20f10-f92b-4919-b80d-519ad1120294\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-khvpj" Oct 30 21:40:05.859737 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:05.859707 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39e20f10-f92b-4919-b80d-519ad1120294-credential-socket\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-khvpj\" (UID: \"39e20f10-f92b-4919-b80d-519ad1120294\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-khvpj" Oct 30 21:40:05.859737 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:05.859729 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39e20f10-f92b-4919-b80d-519ad1120294-workload-certs\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-khvpj\" (UID: \"39e20f10-f92b-4919-b80d-519ad1120294\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-khvpj" Oct 30 21:40:05.860116 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:05.860083 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39e20f10-f92b-4919-b80d-519ad1120294-workload-socket\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-khvpj\" (UID: \"39e20f10-f92b-4919-b80d-519ad1120294\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-khvpj" Oct 30 21:40:05.860197 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:05.860133 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39e20f10-f92b-4919-b80d-519ad1120294-istio-data\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-khvpj\" (UID: \"39e20f10-f92b-4919-b80d-519ad1120294\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-khvpj" Oct 30 21:40:05.860391 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:05.860366 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39e20f10-f92b-4919-b80d-519ad1120294-istiod-ca-cert\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-khvpj\" (UID: \"39e20f10-f92b-4919-b80d-519ad1120294\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-khvpj" Oct 30 21:40:05.862043 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:05.862021 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39e20f10-f92b-4919-b80d-519ad1120294-istio-envoy\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-khvpj\" (UID: \"39e20f10-f92b-4919-b80d-519ad1120294\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-khvpj" Oct 30 21:40:05.862244 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:05.862225 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39e20f10-f92b-4919-b80d-519ad1120294-istio-podinfo\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-khvpj\" (UID: \"39e20f10-f92b-4919-b80d-519ad1120294\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-khvpj" Oct 30 21:40:05.926719 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:05.926680 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jht5g\" (UniqueName: \"kubernetes.io/projected/39e20f10-f92b-4919-b80d-519ad1120294-kube-api-access-jht5g\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-khvpj\" (UID: \"39e20f10-f92b-4919-b80d-519ad1120294\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-khvpj" Oct 30 21:40:05.928443 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:05.928420 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39e20f10-f92b-4919-b80d-519ad1120294-istio-token\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-khvpj\" (UID: \"39e20f10-f92b-4919-b80d-519ad1120294\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-khvpj" Oct 30 21:40:06.029061 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:06.028968 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-khvpj" Oct 30 21:40:06.466074 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:06.466044 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-khvpj"] Oct 30 21:40:06.467210 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:40:06.467183 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39e20f10_f92b_4919_b80d_519ad1120294.slice/crio-7279a9eb1fa29f7e6a0e728ba20cb7e749c2565588b2815f115fdd571bef0741 WatchSource:0}: Error finding container 7279a9eb1fa29f7e6a0e728ba20cb7e749c2565588b2815f115fdd571bef0741: Status 404 returned error can't find the container with id 7279a9eb1fa29f7e6a0e728ba20cb7e749c2565588b2815f115fdd571bef0741 Oct 30 21:40:06.470231 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:06.470196 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:40:06.470333 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:06.470260 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:40:06.470333 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:06.470295 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:40:06.899072 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:06.898165 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7" Oct 30 21:40:06.899072 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:06.898547 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7" Oct 30 21:40:06.900367 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:06.900341 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7" Oct 30 21:40:07.397428 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:07.397395 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-khvpj" event={"ID":"39e20f10-f92b-4919-b80d-519ad1120294","Type":"ContainerStarted","Data":"cf5f69ad3879667f2b7781df6e7f1bf46ba764ead9955350cac6e90f70cb4e15"} Oct 30 21:40:07.397575 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:07.397454 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-khvpj" event={"ID":"39e20f10-f92b-4919-b80d-519ad1120294","Type":"ContainerStarted","Data":"7279a9eb1fa29f7e6a0e728ba20cb7e749c2565588b2815f115fdd571bef0741"} Oct 30 21:40:07.525345 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:07.524040 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-khvpj" podStartSLOduration=2.524016156 podStartE2EDuration="2.524016156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:07.52259862 +0000 UTC m=+991.265119556" watchObservedRunningTime="2026-10-30 21:40:07.524016156 +0000 UTC m=+991.266537091" Oct 30 21:40:07.565298 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:07.565259 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4ff1ba1-j2gkx"] Oct 30 21:40:07.569541 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:07.569515 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4ff1ba1-j2gkx" Oct 30 21:40:07.624921 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:07.624268 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:40:07.624921 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:07.624432 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:40:07.624921 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:07.624532 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 21:40:07.624921 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:07.624613 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:40:07.638082 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:07.638048 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4ff1ba1-j2gkx"] Oct 30 21:40:07.683643 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:07.683554 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f7e0cdb-2273-4bfa-bba8-fa5c86f13f5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4ff1ba1-j2gkx\" (UID: \"5f7e0cdb-2273-4bfa-bba8-fa5c86f13f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4ff1ba1-j2gkx" Oct 30 21:40:07.683643 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:07.683601 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwg82\" (UniqueName: \"kubernetes.io/projected/5f7e0cdb-2273-4bfa-bba8-fa5c86f13f5d-kube-api-access-mwg82\") pod \"maas-api-revoke-sub-b4ff1ba1-j2gkx\" (UID: \"5f7e0cdb-2273-4bfa-bba8-fa5c86f13f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4ff1ba1-j2gkx" Oct 30 21:40:07.683934 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:07.683816 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f7e0cdb-2273-4bfa-bba8-fa5c86f13f5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4ff1ba1-j2gkx\" (UID: \"5f7e0cdb-2273-4bfa-bba8-fa5c86f13f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4ff1ba1-j2gkx" Oct 30 21:40:07.785099 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:07.785062 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f7e0cdb-2273-4bfa-bba8-fa5c86f13f5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4ff1ba1-j2gkx\" (UID: \"5f7e0cdb-2273-4bfa-bba8-fa5c86f13f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4ff1ba1-j2gkx" Oct 30 21:40:07.785309 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:07.785117 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwg82\" (UniqueName: \"kubernetes.io/projected/5f7e0cdb-2273-4bfa-bba8-fa5c86f13f5d-kube-api-access-mwg82\") pod \"maas-api-revoke-sub-b4ff1ba1-j2gkx\" (UID: \"5f7e0cdb-2273-4bfa-bba8-fa5c86f13f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4ff1ba1-j2gkx" Oct 30 21:40:07.785309 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:07.785195 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f7e0cdb-2273-4bfa-bba8-fa5c86f13f5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4ff1ba1-j2gkx\" (UID: \"5f7e0cdb-2273-4bfa-bba8-fa5c86f13f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4ff1ba1-j2gkx" Oct 30 21:40:07.785828 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:07.785808 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f7e0cdb-2273-4bfa-bba8-fa5c86f13f5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4ff1ba1-j2gkx\" (UID: \"5f7e0cdb-2273-4bfa-bba8-fa5c86f13f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4ff1ba1-j2gkx" Oct 30 21:40:07.787494 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:07.787472 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f7e0cdb-2273-4bfa-bba8-fa5c86f13f5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4ff1ba1-j2gkx\" (UID: \"5f7e0cdb-2273-4bfa-bba8-fa5c86f13f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4ff1ba1-j2gkx" Oct 30 21:40:07.847227 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:07.847188 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwg82\" (UniqueName: \"kubernetes.io/projected/5f7e0cdb-2273-4bfa-bba8-fa5c86f13f5d-kube-api-access-mwg82\") pod \"maas-api-revoke-sub-b4ff1ba1-j2gkx\" (UID: \"5f7e0cdb-2273-4bfa-bba8-fa5c86f13f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4ff1ba1-j2gkx" Oct 30 21:40:07.884138 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:07.884101 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4ff1ba1-j2gkx" Oct 30 21:40:08.030083 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:08.030048 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-khvpj" Oct 30 21:40:08.032260 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:08.031928 5748 patch_prober.go:29] interesting pod/e2e-worker-w5-c7d477-openshift-default-776595695-khvpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 21:40:08.032260 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:08.031983 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-khvpj" podUID="39e20f10-f92b-4919-b80d-519ad1120294" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 21:40:08.295525 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:08.295494 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4ff1ba1-j2gkx"] Oct 30 21:40:08.296632 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:40:08.296608 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f7e0cdb_2273_4bfa_bba8_fa5c86f13f5d.slice/crio-eee1884f0b49e19d6fb13e39dddc14c4f2d38ba5890c9d65b1b1be2f95f75bf2 WatchSource:0}: Error finding container eee1884f0b49e19d6fb13e39dddc14c4f2d38ba5890c9d65b1b1be2f95f75bf2: Status 404 returned error can't find the container with id eee1884f0b49e19d6fb13e39dddc14c4f2d38ba5890c9d65b1b1be2f95f75bf2 Oct 30 21:40:08.433756 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:08.433720 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4ff1ba1-j2gkx" event={"ID":"5f7e0cdb-2273-4bfa-bba8-fa5c86f13f5d","Type":"ContainerStarted","Data":"eee1884f0b49e19d6fb13e39dddc14c4f2d38ba5890c9d65b1b1be2f95f75bf2"} Oct 30 21:40:09.029782 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:09.029700 5748 patch_prober.go:29] interesting pod/e2e-worker-w5-c7d477-openshift-default-776595695-khvpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 21:40:09.029782 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:09.029761 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-khvpj" podUID="39e20f10-f92b-4919-b80d-519ad1120294" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 21:40:09.439846 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:09.439810 5748 generic.go:356] "Generic (PLEG): container finished" podID="5f7e0cdb-2273-4bfa-bba8-fa5c86f13f5d" containerID="fcb05ff7dc55df4ac2a9356ec89369b5d3876b2dc0e75d958ba5f1f88aac4065" exitCode=0 Oct 30 21:40:09.440257 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:09.439858 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4ff1ba1-j2gkx" event={"ID":"5f7e0cdb-2273-4bfa-bba8-fa5c86f13f5d","Type":"ContainerDied","Data":"fcb05ff7dc55df4ac2a9356ec89369b5d3876b2dc0e75d958ba5f1f88aac4065"} Oct 30 21:40:10.030190 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:10.030151 5748 patch_prober.go:29] interesting pod/e2e-worker-w5-c7d477-openshift-default-776595695-khvpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 21:40:10.030353 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:10.030242 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-khvpj" podUID="39e20f10-f92b-4919-b80d-519ad1120294" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 21:40:10.653081 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:10.653050 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4ff1ba1-j2gkx" Oct 30 21:40:10.740313 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:10.740208 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f7e0cdb-2273-4bfa-bba8-fa5c86f13f5d-kube-api-access-mwg82\" (UniqueName: \"kubernetes.io/projected/5f7e0cdb-2273-4bfa-bba8-fa5c86f13f5d-kube-api-access-mwg82\") pod \"5f7e0cdb-2273-4bfa-bba8-fa5c86f13f5d\" (UID: \"5f7e0cdb-2273-4bfa-bba8-fa5c86f13f5d\") " Oct 30 21:40:10.740477 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:10.740317 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f7e0cdb-2273-4bfa-bba8-fa5c86f13f5d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f7e0cdb-2273-4bfa-bba8-fa5c86f13f5d-maas-api-xks-ca\") pod \"5f7e0cdb-2273-4bfa-bba8-fa5c86f13f5d\" (UID: \"5f7e0cdb-2273-4bfa-bba8-fa5c86f13f5d\") " Oct 30 21:40:10.740477 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:10.740366 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f7e0cdb-2273-4bfa-bba8-fa5c86f13f5d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f7e0cdb-2273-4bfa-bba8-fa5c86f13f5d-maas-api-openshift-ca\") pod \"5f7e0cdb-2273-4bfa-bba8-fa5c86f13f5d\" (UID: \"5f7e0cdb-2273-4bfa-bba8-fa5c86f13f5d\") " Oct 30 21:40:10.741083 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:10.741034 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f7e0cdb-2273-4bfa-bba8-fa5c86f13f5d-maas-api-openshift-ca" pod "5f7e0cdb-2273-4bfa-bba8-fa5c86f13f5d" (UID: "5f7e0cdb-2273-4bfa-bba8-fa5c86f13f5d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:40:10.745091 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:10.745060 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f7e0cdb-2273-4bfa-bba8-fa5c86f13f5d-kube-api-access-mwg82" pod "5f7e0cdb-2273-4bfa-bba8-fa5c86f13f5d" (UID: "5f7e0cdb-2273-4bfa-bba8-fa5c86f13f5d"). InnerVolumeSpecName "kube-api-access-mwg82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:40:10.745406 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:10.745385 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f7e0cdb-2273-4bfa-bba8-fa5c86f13f5d-maas-api-xks-ca" pod "5f7e0cdb-2273-4bfa-bba8-fa5c86f13f5d" (UID: "5f7e0cdb-2273-4bfa-bba8-fa5c86f13f5d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:40:10.842041 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:10.841999 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwg82\" (UniqueName: \"kubernetes.io/projected/5f7e0cdb-2273-4bfa-bba8-fa5c86f13f5d-kube-api-access-mwg82\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:40:10.842041 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:10.842037 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f7e0cdb-2273-4bfa-bba8-fa5c86f13f5d-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:40:10.842302 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:10.842053 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f7e0cdb-2273-4bfa-bba8-fa5c86f13f5d-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:40:11.036877 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:11.036337 5748 patch_prober.go:29] interesting pod/e2e-worker-w5-c7d477-openshift-default-776595695-khvpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 21:40:11.036877 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:11.036420 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-khvpj" podUID="39e20f10-f92b-4919-b80d-519ad1120294" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 21:40:11.489579 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:11.488070 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4ff1ba1-j2gkx" event={"ID":"5f7e0cdb-2273-4bfa-bba8-fa5c86f13f5d","Type":"ContainerDied","Data":"eee1884f0b49e19d6fb13e39dddc14c4f2d38ba5890c9d65b1b1be2f95f75bf2"} Oct 30 21:40:11.489579 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:11.488142 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eee1884f0b49e19d6fb13e39dddc14c4f2d38ba5890c9d65b1b1be2f95f75bf2" Oct 30 21:40:11.489579 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:11.488219 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4ff1ba1-j2gkx" Oct 30 21:40:12.029437 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:12.029347 5748 patch_prober.go:29] interesting pod/e2e-worker-w5-c7d477-openshift-default-776595695-khvpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 21:40:12.029437 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:12.029420 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-khvpj" podUID="39e20f10-f92b-4919-b80d-519ad1120294" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 21:40:13.030368 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:13.030330 5748 patch_prober.go:29] interesting pod/e2e-worker-w5-c7d477-openshift-default-776595695-khvpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 21:40:13.030807 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:13.030388 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-khvpj" podUID="39e20f10-f92b-4919-b80d-519ad1120294" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 21:40:14.136637 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:14.136604 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-khvpj" Oct 30 21:40:14.137139 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:14.136883 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-khvpj" Oct 30 21:40:14.138049 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:14.138031 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-khvpj" Oct 30 21:40:16.845000 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:16.844965 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6be7c7d8-q65x9"] Oct 30 21:40:16.845465 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:16.845451 5748 cpu_manager.go:414] "containerMap: removing container" podUID="5f7e0cdb-2273-4bfa-bba8-fa5c86f13f5d" containerName="revoke-keys" Oct 30 21:40:16.845509 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:16.845466 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f7e0cdb-2273-4bfa-bba8-fa5c86f13f5d" containerName="revoke-keys" Oct 30 21:40:16.845572 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:16.845564 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f7e0cdb-2273-4bfa-bba8-fa5c86f13f5d" containerName="revoke-keys" Oct 30 21:40:16.849220 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:16.849202 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be7c7d8-q65x9" Oct 30 21:40:16.929655 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:16.929618 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvlxg\" (UniqueName: \"kubernetes.io/projected/d05672e8-e4f1-465c-acca-1c7e0c4858bf-kube-api-access-vvlxg\") pod \"maas-api-revoke-sub-6be7c7d8-q65x9\" (UID: \"d05672e8-e4f1-465c-acca-1c7e0c4858bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be7c7d8-q65x9" Oct 30 21:40:16.929824 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:16.929674 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d05672e8-e4f1-465c-acca-1c7e0c4858bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6be7c7d8-q65x9\" (UID: \"d05672e8-e4f1-465c-acca-1c7e0c4858bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be7c7d8-q65x9" Oct 30 21:40:16.929824 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:16.929734 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d05672e8-e4f1-465c-acca-1c7e0c4858bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6be7c7d8-q65x9\" (UID: \"d05672e8-e4f1-465c-acca-1c7e0c4858bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be7c7d8-q65x9" Oct 30 21:40:16.954201 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:16.954171 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:40:16.954201 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:16.954184 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 21:40:16.954201 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:16.954211 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:40:16.954443 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:16.954241 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:40:16.957355 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:16.957328 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6be7c7d8-q65x9"] Oct 30 21:40:17.031386 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:17.031349 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvlxg\" (UniqueName: \"kubernetes.io/projected/d05672e8-e4f1-465c-acca-1c7e0c4858bf-kube-api-access-vvlxg\") pod \"maas-api-revoke-sub-6be7c7d8-q65x9\" (UID: \"d05672e8-e4f1-465c-acca-1c7e0c4858bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be7c7d8-q65x9" Oct 30 21:40:17.031386 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:17.031388 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d05672e8-e4f1-465c-acca-1c7e0c4858bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6be7c7d8-q65x9\" (UID: \"d05672e8-e4f1-465c-acca-1c7e0c4858bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be7c7d8-q65x9" Oct 30 21:40:17.031602 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:17.031421 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d05672e8-e4f1-465c-acca-1c7e0c4858bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6be7c7d8-q65x9\" (UID: \"d05672e8-e4f1-465c-acca-1c7e0c4858bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be7c7d8-q65x9" Oct 30 21:40:17.032115 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:17.032095 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d05672e8-e4f1-465c-acca-1c7e0c4858bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6be7c7d8-q65x9\" (UID: \"d05672e8-e4f1-465c-acca-1c7e0c4858bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be7c7d8-q65x9" Oct 30 21:40:17.033825 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:17.033800 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d05672e8-e4f1-465c-acca-1c7e0c4858bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6be7c7d8-q65x9\" (UID: \"d05672e8-e4f1-465c-acca-1c7e0c4858bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be7c7d8-q65x9" Oct 30 21:40:17.097109 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:17.097036 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvlxg\" (UniqueName: \"kubernetes.io/projected/d05672e8-e4f1-465c-acca-1c7e0c4858bf-kube-api-access-vvlxg\") pod \"maas-api-revoke-sub-6be7c7d8-q65x9\" (UID: \"d05672e8-e4f1-465c-acca-1c7e0c4858bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be7c7d8-q65x9" Oct 30 21:40:17.162028 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:17.161992 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be7c7d8-q65x9" Oct 30 21:40:17.556379 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:40:17.556348 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd05672e8_e4f1_465c_acca_1c7e0c4858bf.slice/crio-007822784b58c4d6a6dd242a3c54f78190aab10690a98cdc144142ae6f6b76ad WatchSource:0}: Error finding container 007822784b58c4d6a6dd242a3c54f78190aab10690a98cdc144142ae6f6b76ad: Status 404 returned error can't find the container with id 007822784b58c4d6a6dd242a3c54f78190aab10690a98cdc144142ae6f6b76ad Oct 30 21:40:17.556626 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:17.556604 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6be7c7d8-q65x9"] Oct 30 21:40:18.534395 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:18.534354 5748 generic.go:356] "Generic (PLEG): container finished" podID="d05672e8-e4f1-465c-acca-1c7e0c4858bf" containerID="ad7215a71168bd6f91c9f16b06d9d0ed85af38dd506472da47ff2d8045180a1e" exitCode=0 Oct 30 21:40:18.534801 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:18.534438 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be7c7d8-q65x9" event={"ID":"d05672e8-e4f1-465c-acca-1c7e0c4858bf","Type":"ContainerDied","Data":"ad7215a71168bd6f91c9f16b06d9d0ed85af38dd506472da47ff2d8045180a1e"} Oct 30 21:40:18.534801 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:18.534513 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be7c7d8-q65x9" event={"ID":"d05672e8-e4f1-465c-acca-1c7e0c4858bf","Type":"ContainerStarted","Data":"007822784b58c4d6a6dd242a3c54f78190aab10690a98cdc144142ae6f6b76ad"} Oct 30 21:40:19.681193 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:19.681168 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be7c7d8-q65x9" Oct 30 21:40:19.762137 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:19.762107 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d05672e8-e4f1-465c-acca-1c7e0c4858bf-kube-api-access-vvlxg\" (UniqueName: \"kubernetes.io/projected/d05672e8-e4f1-465c-acca-1c7e0c4858bf-kube-api-access-vvlxg\") pod \"d05672e8-e4f1-465c-acca-1c7e0c4858bf\" (UID: \"d05672e8-e4f1-465c-acca-1c7e0c4858bf\") " Oct 30 21:40:19.762297 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:19.762198 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d05672e8-e4f1-465c-acca-1c7e0c4858bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d05672e8-e4f1-465c-acca-1c7e0c4858bf-maas-api-xks-ca\") pod \"d05672e8-e4f1-465c-acca-1c7e0c4858bf\" (UID: \"d05672e8-e4f1-465c-acca-1c7e0c4858bf\") " Oct 30 21:40:19.762297 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:19.762227 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d05672e8-e4f1-465c-acca-1c7e0c4858bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d05672e8-e4f1-465c-acca-1c7e0c4858bf-maas-api-openshift-ca\") pod \"d05672e8-e4f1-465c-acca-1c7e0c4858bf\" (UID: \"d05672e8-e4f1-465c-acca-1c7e0c4858bf\") " Oct 30 21:40:19.762655 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:19.762627 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d05672e8-e4f1-465c-acca-1c7e0c4858bf-maas-api-openshift-ca" pod "d05672e8-e4f1-465c-acca-1c7e0c4858bf" (UID: "d05672e8-e4f1-465c-acca-1c7e0c4858bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:40:19.762815 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:19.762800 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d05672e8-e4f1-465c-acca-1c7e0c4858bf-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:40:19.764245 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:19.764217 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d05672e8-e4f1-465c-acca-1c7e0c4858bf-kube-api-access-vvlxg" pod "d05672e8-e4f1-465c-acca-1c7e0c4858bf" (UID: "d05672e8-e4f1-465c-acca-1c7e0c4858bf"). InnerVolumeSpecName "kube-api-access-vvlxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:40:19.764364 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:19.764287 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d05672e8-e4f1-465c-acca-1c7e0c4858bf-maas-api-xks-ca" pod "d05672e8-e4f1-465c-acca-1c7e0c4858bf" (UID: "d05672e8-e4f1-465c-acca-1c7e0c4858bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:40:19.864575 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:19.864504 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvlxg\" (UniqueName: \"kubernetes.io/projected/d05672e8-e4f1-465c-acca-1c7e0c4858bf-kube-api-access-vvlxg\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:40:19.864575 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:19.864530 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d05672e8-e4f1-465c-acca-1c7e0c4858bf-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:40:20.550006 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:20.549971 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be7c7d8-q65x9" event={"ID":"d05672e8-e4f1-465c-acca-1c7e0c4858bf","Type":"ContainerDied","Data":"007822784b58c4d6a6dd242a3c54f78190aab10690a98cdc144142ae6f6b76ad"} Oct 30 21:40:20.550160 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:20.550008 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be7c7d8-q65x9" Oct 30 21:40:20.550160 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:20.550020 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="007822784b58c4d6a6dd242a3c54f78190aab10690a98cdc144142ae6f6b76ad" Oct 30 21:40:26.334344 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:26.334307 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-d68156/distinct-w3-d68156-kserve-565894fdb4-85t7z"] Oct 30 21:40:26.334861 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:26.334846 5748 cpu_manager.go:414] "containerMap: removing container" podUID="d05672e8-e4f1-465c-acca-1c7e0c4858bf" containerName="revoke-keys" Oct 30 21:40:26.334933 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:26.334863 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d05672e8-e4f1-465c-acca-1c7e0c4858bf" containerName="revoke-keys" Oct 30 21:40:26.334999 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:26.334989 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="d05672e8-e4f1-465c-acca-1c7e0c4858bf" containerName="revoke-keys" Oct 30 21:40:26.338676 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:26.338654 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d68156/distinct-w3-d68156-kserve-565894fdb4-85t7z" Oct 30 21:40:26.439832 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:26.439781 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2bhb\" (UniqueName: \"kubernetes.io/projected/13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe-kube-api-access-q2bhb\") pod \"distinct-w3-d68156-kserve-565894fdb4-85t7z\" (UID: \"13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe\") " pod="e2e-models-e2e-worker-w3-d68156/distinct-w3-d68156-kserve-565894fdb4-85t7z" Oct 30 21:40:26.440042 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:26.439845 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe-dshm\") pod \"distinct-w3-d68156-kserve-565894fdb4-85t7z\" (UID: \"13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe\") " pod="e2e-models-e2e-worker-w3-d68156/distinct-w3-d68156-kserve-565894fdb4-85t7z" Oct 30 21:40:26.440042 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:26.439981 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe-tmp-dir\") pod \"distinct-w3-d68156-kserve-565894fdb4-85t7z\" (UID: \"13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe\") " pod="e2e-models-e2e-worker-w3-d68156/distinct-w3-d68156-kserve-565894fdb4-85t7z" Oct 30 21:40:26.440218 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:26.440190 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe-model-cache\") pod \"distinct-w3-d68156-kserve-565894fdb4-85t7z\" (UID: \"13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe\") " pod="e2e-models-e2e-worker-w3-d68156/distinct-w3-d68156-kserve-565894fdb4-85t7z" Oct 30 21:40:26.440301 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:26.440267 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe-home\") pod \"distinct-w3-d68156-kserve-565894fdb4-85t7z\" (UID: \"13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe\") " pod="e2e-models-e2e-worker-w3-d68156/distinct-w3-d68156-kserve-565894fdb4-85t7z" Oct 30 21:40:26.440729 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:26.440707 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe-tls-certs\") pod \"distinct-w3-d68156-kserve-565894fdb4-85t7z\" (UID: \"13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe\") " pod="e2e-models-e2e-worker-w3-d68156/distinct-w3-d68156-kserve-565894fdb4-85t7z" Oct 30 21:40:26.442861 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:26.442844 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-d68156\"/\"distinct-w3-d68156-kserve-self-signed-certs\"" Oct 30 21:40:26.446066 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:26.446047 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d68156/distinct-w3-d68156-kserve-565894fdb4-85t7z"] Oct 30 21:40:26.541948 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:26.541892 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe-tls-certs\") pod \"distinct-w3-d68156-kserve-565894fdb4-85t7z\" (UID: \"13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe\") " pod="e2e-models-e2e-worker-w3-d68156/distinct-w3-d68156-kserve-565894fdb4-85t7z" Oct 30 21:40:26.542129 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:26.541962 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2bhb\" (UniqueName: \"kubernetes.io/projected/13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe-kube-api-access-q2bhb\") pod \"distinct-w3-d68156-kserve-565894fdb4-85t7z\" (UID: \"13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe\") " pod="e2e-models-e2e-worker-w3-d68156/distinct-w3-d68156-kserve-565894fdb4-85t7z" Oct 30 21:40:26.542129 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:26.541988 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe-dshm\") pod \"distinct-w3-d68156-kserve-565894fdb4-85t7z\" (UID: \"13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe\") " pod="e2e-models-e2e-worker-w3-d68156/distinct-w3-d68156-kserve-565894fdb4-85t7z" Oct 30 21:40:26.542129 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:26.542065 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe-tmp-dir\") pod \"distinct-w3-d68156-kserve-565894fdb4-85t7z\" (UID: \"13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe\") " pod="e2e-models-e2e-worker-w3-d68156/distinct-w3-d68156-kserve-565894fdb4-85t7z" Oct 30 21:40:26.542270 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:26.542215 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe-model-cache\") pod \"distinct-w3-d68156-kserve-565894fdb4-85t7z\" (UID: \"13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe\") " pod="e2e-models-e2e-worker-w3-d68156/distinct-w3-d68156-kserve-565894fdb4-85t7z" Oct 30 21:40:26.542327 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:26.542271 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe-home\") pod \"distinct-w3-d68156-kserve-565894fdb4-85t7z\" (UID: \"13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe\") " pod="e2e-models-e2e-worker-w3-d68156/distinct-w3-d68156-kserve-565894fdb4-85t7z" Oct 30 21:40:26.542466 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:26.542441 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe-tmp-dir\") pod \"distinct-w3-d68156-kserve-565894fdb4-85t7z\" (UID: \"13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe\") " pod="e2e-models-e2e-worker-w3-d68156/distinct-w3-d68156-kserve-565894fdb4-85t7z" Oct 30 21:40:26.542626 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:26.542604 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe-model-cache\") pod \"distinct-w3-d68156-kserve-565894fdb4-85t7z\" (UID: \"13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe\") " pod="e2e-models-e2e-worker-w3-d68156/distinct-w3-d68156-kserve-565894fdb4-85t7z" Oct 30 21:40:26.542626 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:26.542618 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe-home\") pod \"distinct-w3-d68156-kserve-565894fdb4-85t7z\" (UID: \"13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe\") " pod="e2e-models-e2e-worker-w3-d68156/distinct-w3-d68156-kserve-565894fdb4-85t7z" Oct 30 21:40:26.544257 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:26.544231 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe-dshm\") pod \"distinct-w3-d68156-kserve-565894fdb4-85t7z\" (UID: \"13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe\") " pod="e2e-models-e2e-worker-w3-d68156/distinct-w3-d68156-kserve-565894fdb4-85t7z" Oct 30 21:40:26.544667 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:26.544639 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe-tls-certs\") pod \"distinct-w3-d68156-kserve-565894fdb4-85t7z\" (UID: \"13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe\") " pod="e2e-models-e2e-worker-w3-d68156/distinct-w3-d68156-kserve-565894fdb4-85t7z" Oct 30 21:40:26.606994 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:26.606915 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2bhb\" (UniqueName: \"kubernetes.io/projected/13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe-kube-api-access-q2bhb\") pod \"distinct-w3-d68156-kserve-565894fdb4-85t7z\" (UID: \"13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe\") " pod="e2e-models-e2e-worker-w3-d68156/distinct-w3-d68156-kserve-565894fdb4-85t7z" Oct 30 21:40:26.650856 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:26.650813 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d68156/distinct-w3-d68156-kserve-565894fdb4-85t7z" Oct 30 21:40:27.044399 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:27.044374 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d68156/distinct-w3-d68156-kserve-565894fdb4-85t7z"] Oct 30 21:40:27.045501 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:40:27.045467 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13ec4fff_3e1b_42c2_83dd_4d23c6af6cbe.slice/crio-ad45feefa4e12d4aa7b883bd0eb899f9bb87b65a1842b1c349125bbe73137f0c WatchSource:0}: Error finding container ad45feefa4e12d4aa7b883bd0eb899f9bb87b65a1842b1c349125bbe73137f0c: Status 404 returned error can't find the container with id ad45feefa4e12d4aa7b883bd0eb899f9bb87b65a1842b1c349125bbe73137f0c Oct 30 21:40:27.594554 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:27.594511 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d68156/distinct-w3-d68156-kserve-565894fdb4-85t7z" event={"ID":"13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe","Type":"ContainerStarted","Data":"486949e0219addc738f87a366f0ea90c243baa9a9c6cf4bf9ce86aed48906f0f"} Oct 30 21:40:27.594961 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:27.594566 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d68156/distinct-w3-d68156-kserve-565894fdb4-85t7z" event={"ID":"13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe","Type":"ContainerStarted","Data":"ad45feefa4e12d4aa7b883bd0eb899f9bb87b65a1842b1c349125bbe73137f0c"} Oct 30 21:40:27.651111 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:27.651079 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d68156/distinct-w3-d68156-kserve-565894fdb4-85t7z" Oct 30 21:40:27.714208 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:27.714150 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-d68156/distinct-w3-d68156-kserve-565894fdb4-85t7z" podStartSLOduration=1.714136111 podStartE2EDuration="1.714136111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:27.71282109 +0000 UTC m=+1011.455342027" watchObservedRunningTime="2026-10-30 21:40:27.714136111 +0000 UTC m=+1011.456657045" Oct 30 21:40:30.155232 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:40:30.149256 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:40:36.654279 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:36.654234 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-d68156/distinct-w3-d68156-kserve-565894fdb4-85t7z" Oct 30 21:40:36.667459 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:36.667432 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-d68156/distinct-w3-d68156-kserve-565894fdb4-85t7z" Oct 30 21:40:37.662269 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:40:37.662237 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d68156/distinct-w3-d68156-kserve-565894fdb4-85t7z" Oct 30 21:41:00.145997 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:41:00.145962 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:41:01.422741 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:01.422706 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-c7d477/distinct-w5-c7d477-kserve-554d7f974d-6zw6s"] Oct 30 21:41:01.427036 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:01.427015 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c7d477/distinct-w5-c7d477-kserve-554d7f974d-6zw6s" Oct 30 21:41:01.437403 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:01.437377 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09d39cc0-b68d-48fd-918c-7d2518f384aa-tmp-dir\") pod \"distinct-w5-c7d477-kserve-554d7f974d-6zw6s\" (UID: \"09d39cc0-b68d-48fd-918c-7d2518f384aa\") " pod="e2e-models-e2e-worker-w5-c7d477/distinct-w5-c7d477-kserve-554d7f974d-6zw6s" Oct 30 21:41:01.437552 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:01.437415 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09d39cc0-b68d-48fd-918c-7d2518f384aa-tls-certs\") pod \"distinct-w5-c7d477-kserve-554d7f974d-6zw6s\" (UID: \"09d39cc0-b68d-48fd-918c-7d2518f384aa\") " pod="e2e-models-e2e-worker-w5-c7d477/distinct-w5-c7d477-kserve-554d7f974d-6zw6s" Oct 30 21:41:01.437552 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:01.437442 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09d39cc0-b68d-48fd-918c-7d2518f384aa-home\") pod \"distinct-w5-c7d477-kserve-554d7f974d-6zw6s\" (UID: \"09d39cc0-b68d-48fd-918c-7d2518f384aa\") " pod="e2e-models-e2e-worker-w5-c7d477/distinct-w5-c7d477-kserve-554d7f974d-6zw6s" Oct 30 21:41:01.437672 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:01.437578 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09d39cc0-b68d-48fd-918c-7d2518f384aa-dshm\") pod \"distinct-w5-c7d477-kserve-554d7f974d-6zw6s\" (UID: \"09d39cc0-b68d-48fd-918c-7d2518f384aa\") " pod="e2e-models-e2e-worker-w5-c7d477/distinct-w5-c7d477-kserve-554d7f974d-6zw6s" Oct 30 21:41:01.437672 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:01.437650 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09d39cc0-b68d-48fd-918c-7d2518f384aa-model-cache\") pod \"distinct-w5-c7d477-kserve-554d7f974d-6zw6s\" (UID: \"09d39cc0-b68d-48fd-918c-7d2518f384aa\") " pod="e2e-models-e2e-worker-w5-c7d477/distinct-w5-c7d477-kserve-554d7f974d-6zw6s" Oct 30 21:41:01.437804 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:01.437719 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mkjj\" (UniqueName: \"kubernetes.io/projected/09d39cc0-b68d-48fd-918c-7d2518f384aa-kube-api-access-2mkjj\") pod \"distinct-w5-c7d477-kserve-554d7f974d-6zw6s\" (UID: \"09d39cc0-b68d-48fd-918c-7d2518f384aa\") " pod="e2e-models-e2e-worker-w5-c7d477/distinct-w5-c7d477-kserve-554d7f974d-6zw6s" Oct 30 21:41:01.533684 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:01.533645 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-c7d477\"/\"distinct-w5-c7d477-kserve-self-signed-certs\"" Oct 30 21:41:01.536701 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:01.536675 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-c7d477/distinct-w5-c7d477-kserve-554d7f974d-6zw6s"] Oct 30 21:41:01.539062 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:01.539039 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09d39cc0-b68d-48fd-918c-7d2518f384aa-tmp-dir\") pod \"distinct-w5-c7d477-kserve-554d7f974d-6zw6s\" (UID: \"09d39cc0-b68d-48fd-918c-7d2518f384aa\") " pod="e2e-models-e2e-worker-w5-c7d477/distinct-w5-c7d477-kserve-554d7f974d-6zw6s" Oct 30 21:41:01.539123 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:01.539077 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09d39cc0-b68d-48fd-918c-7d2518f384aa-tls-certs\") pod \"distinct-w5-c7d477-kserve-554d7f974d-6zw6s\" (UID: \"09d39cc0-b68d-48fd-918c-7d2518f384aa\") " pod="e2e-models-e2e-worker-w5-c7d477/distinct-w5-c7d477-kserve-554d7f974d-6zw6s" Oct 30 21:41:01.539123 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:01.539103 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09d39cc0-b68d-48fd-918c-7d2518f384aa-home\") pod \"distinct-w5-c7d477-kserve-554d7f974d-6zw6s\" (UID: \"09d39cc0-b68d-48fd-918c-7d2518f384aa\") " pod="e2e-models-e2e-worker-w5-c7d477/distinct-w5-c7d477-kserve-554d7f974d-6zw6s" Oct 30 21:41:01.539204 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:01.539134 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09d39cc0-b68d-48fd-918c-7d2518f384aa-dshm\") pod \"distinct-w5-c7d477-kserve-554d7f974d-6zw6s\" (UID: \"09d39cc0-b68d-48fd-918c-7d2518f384aa\") " pod="e2e-models-e2e-worker-w5-c7d477/distinct-w5-c7d477-kserve-554d7f974d-6zw6s" Oct 30 21:41:01.539204 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:01.539157 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09d39cc0-b68d-48fd-918c-7d2518f384aa-model-cache\") pod \"distinct-w5-c7d477-kserve-554d7f974d-6zw6s\" (UID: \"09d39cc0-b68d-48fd-918c-7d2518f384aa\") " pod="e2e-models-e2e-worker-w5-c7d477/distinct-w5-c7d477-kserve-554d7f974d-6zw6s" Oct 30 21:41:01.539356 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:01.539209 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mkjj\" (UniqueName: \"kubernetes.io/projected/09d39cc0-b68d-48fd-918c-7d2518f384aa-kube-api-access-2mkjj\") pod \"distinct-w5-c7d477-kserve-554d7f974d-6zw6s\" (UID: \"09d39cc0-b68d-48fd-918c-7d2518f384aa\") " pod="e2e-models-e2e-worker-w5-c7d477/distinct-w5-c7d477-kserve-554d7f974d-6zw6s" Oct 30 21:41:01.539562 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:01.539530 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09d39cc0-b68d-48fd-918c-7d2518f384aa-tmp-dir\") pod \"distinct-w5-c7d477-kserve-554d7f974d-6zw6s\" (UID: \"09d39cc0-b68d-48fd-918c-7d2518f384aa\") " pod="e2e-models-e2e-worker-w5-c7d477/distinct-w5-c7d477-kserve-554d7f974d-6zw6s" Oct 30 21:41:01.539675 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:01.539593 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09d39cc0-b68d-48fd-918c-7d2518f384aa-home\") pod \"distinct-w5-c7d477-kserve-554d7f974d-6zw6s\" (UID: \"09d39cc0-b68d-48fd-918c-7d2518f384aa\") " pod="e2e-models-e2e-worker-w5-c7d477/distinct-w5-c7d477-kserve-554d7f974d-6zw6s" Oct 30 21:41:01.539675 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:01.539639 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09d39cc0-b68d-48fd-918c-7d2518f384aa-model-cache\") pod \"distinct-w5-c7d477-kserve-554d7f974d-6zw6s\" (UID: \"09d39cc0-b68d-48fd-918c-7d2518f384aa\") " pod="e2e-models-e2e-worker-w5-c7d477/distinct-w5-c7d477-kserve-554d7f974d-6zw6s" Oct 30 21:41:01.541417 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:01.541391 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09d39cc0-b68d-48fd-918c-7d2518f384aa-dshm\") pod \"distinct-w5-c7d477-kserve-554d7f974d-6zw6s\" (UID: \"09d39cc0-b68d-48fd-918c-7d2518f384aa\") " pod="e2e-models-e2e-worker-w5-c7d477/distinct-w5-c7d477-kserve-554d7f974d-6zw6s" Oct 30 21:41:01.541692 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:01.541673 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09d39cc0-b68d-48fd-918c-7d2518f384aa-tls-certs\") pod \"distinct-w5-c7d477-kserve-554d7f974d-6zw6s\" (UID: \"09d39cc0-b68d-48fd-918c-7d2518f384aa\") " pod="e2e-models-e2e-worker-w5-c7d477/distinct-w5-c7d477-kserve-554d7f974d-6zw6s" Oct 30 21:41:01.601964 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:01.601929 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mkjj\" (UniqueName: \"kubernetes.io/projected/09d39cc0-b68d-48fd-918c-7d2518f384aa-kube-api-access-2mkjj\") pod \"distinct-w5-c7d477-kserve-554d7f974d-6zw6s\" (UID: \"09d39cc0-b68d-48fd-918c-7d2518f384aa\") " pod="e2e-models-e2e-worker-w5-c7d477/distinct-w5-c7d477-kserve-554d7f974d-6zw6s" Oct 30 21:41:01.740404 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:01.740267 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c7d477/distinct-w5-c7d477-kserve-554d7f974d-6zw6s" Oct 30 21:41:02.131876 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:02.131840 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-c7d477/distinct-w5-c7d477-kserve-554d7f974d-6zw6s"] Oct 30 21:41:02.132479 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:41:02.132444 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09d39cc0_b68d_48fd_918c_7d2518f384aa.slice/crio-310e655363aecc4d77163999c17ad6443cdd05aa943e9555888f9c70fac056b7 WatchSource:0}: Error finding container 310e655363aecc4d77163999c17ad6443cdd05aa943e9555888f9c70fac056b7: Status 404 returned error can't find the container with id 310e655363aecc4d77163999c17ad6443cdd05aa943e9555888f9c70fac056b7 Oct 30 21:41:02.851696 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:02.851657 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c7d477/distinct-w5-c7d477-kserve-554d7f974d-6zw6s" event={"ID":"09d39cc0-b68d-48fd-918c-7d2518f384aa","Type":"ContainerStarted","Data":"5e7bd4ac6f184e2668da1bea983d9cb15db3fb7acc2ff5f204651708c2b53383"} Oct 30 21:41:02.851696 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:02.851699 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c7d477/distinct-w5-c7d477-kserve-554d7f974d-6zw6s" event={"ID":"09d39cc0-b68d-48fd-918c-7d2518f384aa","Type":"ContainerStarted","Data":"310e655363aecc4d77163999c17ad6443cdd05aa943e9555888f9c70fac056b7"} Oct 30 21:41:02.974068 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:02.974011 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-c7d477/distinct-w5-c7d477-kserve-554d7f974d-6zw6s" podStartSLOduration=1.973996472 podStartE2EDuration="1.973996472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:02.971117732 +0000 UTC m=+1046.713638668" watchObservedRunningTime="2026-10-30 21:41:02.973996472 +0000 UTC m=+1046.716517406" Oct 30 21:41:03.740950 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:03.740907 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c7d477/distinct-w5-c7d477-kserve-554d7f974d-6zw6s" Oct 30 21:41:11.740968 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:11.740934 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-c7d477/distinct-w5-c7d477-kserve-554d7f974d-6zw6s" Oct 30 21:41:11.754842 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:11.754810 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-c7d477/distinct-w5-c7d477-kserve-554d7f974d-6zw6s" Oct 30 21:41:11.919674 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:11.919642 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c7d477/distinct-w5-c7d477-kserve-554d7f974d-6zw6s" Oct 30 21:41:16.734275 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:16.734239 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-91874e/distinct-w2-91874e-kserve-58d6bdfc75-bdp5z"] Oct 30 21:41:16.738517 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:16.738496 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-91874e/distinct-w2-91874e-kserve-58d6bdfc75-bdp5z" Oct 30 21:41:16.822165 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:16.822130 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5qp4\" (UniqueName: \"kubernetes.io/projected/cb39b415-f406-47bd-b591-be7e09240f5a-kube-api-access-q5qp4\") pod \"distinct-w2-91874e-kserve-58d6bdfc75-bdp5z\" (UID: \"cb39b415-f406-47bd-b591-be7e09240f5a\") " pod="e2e-models-e2e-worker-w2-91874e/distinct-w2-91874e-kserve-58d6bdfc75-bdp5z" Oct 30 21:41:16.822362 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:16.822252 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb39b415-f406-47bd-b591-be7e09240f5a-model-cache\") pod \"distinct-w2-91874e-kserve-58d6bdfc75-bdp5z\" (UID: \"cb39b415-f406-47bd-b591-be7e09240f5a\") " pod="e2e-models-e2e-worker-w2-91874e/distinct-w2-91874e-kserve-58d6bdfc75-bdp5z" Oct 30 21:41:16.822362 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:16.822297 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb39b415-f406-47bd-b591-be7e09240f5a-dshm\") pod \"distinct-w2-91874e-kserve-58d6bdfc75-bdp5z\" (UID: \"cb39b415-f406-47bd-b591-be7e09240f5a\") " pod="e2e-models-e2e-worker-w2-91874e/distinct-w2-91874e-kserve-58d6bdfc75-bdp5z" Oct 30 21:41:16.822362 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:16.822312 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb39b415-f406-47bd-b591-be7e09240f5a-tls-certs\") pod \"distinct-w2-91874e-kserve-58d6bdfc75-bdp5z\" (UID: \"cb39b415-f406-47bd-b591-be7e09240f5a\") " pod="e2e-models-e2e-worker-w2-91874e/distinct-w2-91874e-kserve-58d6bdfc75-bdp5z" Oct 30 21:41:16.822362 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:16.822334 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb39b415-f406-47bd-b591-be7e09240f5a-tmp-dir\") pod \"distinct-w2-91874e-kserve-58d6bdfc75-bdp5z\" (UID: \"cb39b415-f406-47bd-b591-be7e09240f5a\") " pod="e2e-models-e2e-worker-w2-91874e/distinct-w2-91874e-kserve-58d6bdfc75-bdp5z" Oct 30 21:41:16.822589 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:16.822506 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb39b415-f406-47bd-b591-be7e09240f5a-home\") pod \"distinct-w2-91874e-kserve-58d6bdfc75-bdp5z\" (UID: \"cb39b415-f406-47bd-b591-be7e09240f5a\") " pod="e2e-models-e2e-worker-w2-91874e/distinct-w2-91874e-kserve-58d6bdfc75-bdp5z" Oct 30 21:41:16.844040 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:16.844005 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-91874e/distinct-w2-91874e-kserve-58d6bdfc75-bdp5z"] Oct 30 21:41:16.844618 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:16.844564 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-91874e\"/\"openshift-service-ca.crt\"" Oct 30 21:41:16.844923 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:16.844889 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-91874e\"/\"kube-root-ca.crt\"" Oct 30 21:41:16.845031 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:16.845014 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-91874e\"/\"distinct-w2-91874e-kserve-self-signed-certs\"" Oct 30 21:41:16.845079 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:16.845067 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-91874e\"/\"default-dockercfg-wsbzn\"" Oct 30 21:41:16.923361 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:16.923329 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5qp4\" (UniqueName: \"kubernetes.io/projected/cb39b415-f406-47bd-b591-be7e09240f5a-kube-api-access-q5qp4\") pod \"distinct-w2-91874e-kserve-58d6bdfc75-bdp5z\" (UID: \"cb39b415-f406-47bd-b591-be7e09240f5a\") " pod="e2e-models-e2e-worker-w2-91874e/distinct-w2-91874e-kserve-58d6bdfc75-bdp5z" Oct 30 21:41:16.923541 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:16.923388 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb39b415-f406-47bd-b591-be7e09240f5a-model-cache\") pod \"distinct-w2-91874e-kserve-58d6bdfc75-bdp5z\" (UID: \"cb39b415-f406-47bd-b591-be7e09240f5a\") " pod="e2e-models-e2e-worker-w2-91874e/distinct-w2-91874e-kserve-58d6bdfc75-bdp5z" Oct 30 21:41:16.923541 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:16.923408 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb39b415-f406-47bd-b591-be7e09240f5a-dshm\") pod \"distinct-w2-91874e-kserve-58d6bdfc75-bdp5z\" (UID: \"cb39b415-f406-47bd-b591-be7e09240f5a\") " pod="e2e-models-e2e-worker-w2-91874e/distinct-w2-91874e-kserve-58d6bdfc75-bdp5z" Oct 30 21:41:16.923541 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:16.923427 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb39b415-f406-47bd-b591-be7e09240f5a-tls-certs\") pod \"distinct-w2-91874e-kserve-58d6bdfc75-bdp5z\" (UID: \"cb39b415-f406-47bd-b591-be7e09240f5a\") " pod="e2e-models-e2e-worker-w2-91874e/distinct-w2-91874e-kserve-58d6bdfc75-bdp5z" Oct 30 21:41:16.923541 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:16.923445 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb39b415-f406-47bd-b591-be7e09240f5a-tmp-dir\") pod \"distinct-w2-91874e-kserve-58d6bdfc75-bdp5z\" (UID: \"cb39b415-f406-47bd-b591-be7e09240f5a\") " pod="e2e-models-e2e-worker-w2-91874e/distinct-w2-91874e-kserve-58d6bdfc75-bdp5z" Oct 30 21:41:16.923761 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:16.923586 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb39b415-f406-47bd-b591-be7e09240f5a-home\") pod \"distinct-w2-91874e-kserve-58d6bdfc75-bdp5z\" (UID: \"cb39b415-f406-47bd-b591-be7e09240f5a\") " pod="e2e-models-e2e-worker-w2-91874e/distinct-w2-91874e-kserve-58d6bdfc75-bdp5z" Oct 30 21:41:16.924002 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:16.923976 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb39b415-f406-47bd-b591-be7e09240f5a-model-cache\") pod \"distinct-w2-91874e-kserve-58d6bdfc75-bdp5z\" (UID: \"cb39b415-f406-47bd-b591-be7e09240f5a\") " pod="e2e-models-e2e-worker-w2-91874e/distinct-w2-91874e-kserve-58d6bdfc75-bdp5z" Oct 30 21:41:16.924002 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:16.923995 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb39b415-f406-47bd-b591-be7e09240f5a-tmp-dir\") pod \"distinct-w2-91874e-kserve-58d6bdfc75-bdp5z\" (UID: \"cb39b415-f406-47bd-b591-be7e09240f5a\") " pod="e2e-models-e2e-worker-w2-91874e/distinct-w2-91874e-kserve-58d6bdfc75-bdp5z" Oct 30 21:41:16.924209 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:16.924061 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb39b415-f406-47bd-b591-be7e09240f5a-home\") pod \"distinct-w2-91874e-kserve-58d6bdfc75-bdp5z\" (UID: \"cb39b415-f406-47bd-b591-be7e09240f5a\") " pod="e2e-models-e2e-worker-w2-91874e/distinct-w2-91874e-kserve-58d6bdfc75-bdp5z" Oct 30 21:41:16.925952 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:16.925929 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb39b415-f406-47bd-b591-be7e09240f5a-dshm\") pod \"distinct-w2-91874e-kserve-58d6bdfc75-bdp5z\" (UID: \"cb39b415-f406-47bd-b591-be7e09240f5a\") " pod="e2e-models-e2e-worker-w2-91874e/distinct-w2-91874e-kserve-58d6bdfc75-bdp5z" Oct 30 21:41:16.926148 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:16.926125 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb39b415-f406-47bd-b591-be7e09240f5a-tls-certs\") pod \"distinct-w2-91874e-kserve-58d6bdfc75-bdp5z\" (UID: \"cb39b415-f406-47bd-b591-be7e09240f5a\") " pod="e2e-models-e2e-worker-w2-91874e/distinct-w2-91874e-kserve-58d6bdfc75-bdp5z" Oct 30 21:41:16.985502 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:16.985412 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5qp4\" (UniqueName: \"kubernetes.io/projected/cb39b415-f406-47bd-b591-be7e09240f5a-kube-api-access-q5qp4\") pod \"distinct-w2-91874e-kserve-58d6bdfc75-bdp5z\" (UID: \"cb39b415-f406-47bd-b591-be7e09240f5a\") " pod="e2e-models-e2e-worker-w2-91874e/distinct-w2-91874e-kserve-58d6bdfc75-bdp5z" Oct 30 21:41:17.051609 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:17.051573 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-91874e/distinct-w2-91874e-kserve-58d6bdfc75-bdp5z" Oct 30 21:41:17.466614 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:17.466585 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-91874e/distinct-w2-91874e-kserve-58d6bdfc75-bdp5z"] Oct 30 21:41:17.467705 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:41:17.467677 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb39b415_f406_47bd_b591_be7e09240f5a.slice/crio-8a5a495c5ca6a7db3187d75babf16f4fee50183997a159451e755faaf9d76ac6 WatchSource:0}: Error finding container 8a5a495c5ca6a7db3187d75babf16f4fee50183997a159451e755faaf9d76ac6: Status 404 returned error can't find the container with id 8a5a495c5ca6a7db3187d75babf16f4fee50183997a159451e755faaf9d76ac6 Oct 30 21:41:17.953038 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:17.952988 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-91874e/distinct-w2-91874e-kserve-58d6bdfc75-bdp5z" event={"ID":"cb39b415-f406-47bd-b591-be7e09240f5a","Type":"ContainerStarted","Data":"89d5456791ba0ef1f88716c7b0b4dffed0a13643555e94539d60efae8876768d"} Oct 30 21:41:17.953411 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:17.953056 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-91874e/distinct-w2-91874e-kserve-58d6bdfc75-bdp5z" event={"ID":"cb39b415-f406-47bd-b591-be7e09240f5a","Type":"ContainerStarted","Data":"8a5a495c5ca6a7db3187d75babf16f4fee50183997a159451e755faaf9d76ac6"} Oct 30 21:41:18.052722 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:18.052693 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-91874e/distinct-w2-91874e-kserve-58d6bdfc75-bdp5z" Oct 30 21:41:18.074852 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:18.074797 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-91874e/distinct-w2-91874e-kserve-58d6bdfc75-bdp5z" podStartSLOduration=2.074784605 podStartE2EDuration="2.074784605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:18.071572652 +0000 UTC m=+1061.814093588" watchObservedRunningTime="2026-10-30 21:41:18.074784605 +0000 UTC m=+1061.817305539" Oct 30 21:41:21.066301 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:21.066262 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m"] Oct 30 21:41:21.070249 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:21.070220 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m" Oct 30 21:41:21.166851 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:21.166816 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m"] Oct 30 21:41:21.175957 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:21.175918 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39fe3952-ddaf-4858-af83-458d1c01f0e8-workload-socket\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m\" (UID: \"39fe3952-ddaf-4858-af83-458d1c01f0e8\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m" Oct 30 21:41:21.176122 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:21.176041 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39fe3952-ddaf-4858-af83-458d1c01f0e8-credential-socket\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m\" (UID: \"39fe3952-ddaf-4858-af83-458d1c01f0e8\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m" Oct 30 21:41:21.176122 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:21.176083 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqc2m\" (UniqueName: \"kubernetes.io/projected/39fe3952-ddaf-4858-af83-458d1c01f0e8-kube-api-access-tqc2m\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m\" (UID: \"39fe3952-ddaf-4858-af83-458d1c01f0e8\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m" Oct 30 21:41:21.176292 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:21.176269 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39fe3952-ddaf-4858-af83-458d1c01f0e8-istio-token\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m\" (UID: \"39fe3952-ddaf-4858-af83-458d1c01f0e8\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m" Oct 30 21:41:21.176342 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:21.176330 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39fe3952-ddaf-4858-af83-458d1c01f0e8-istio-data\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m\" (UID: \"39fe3952-ddaf-4858-af83-458d1c01f0e8\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m" Oct 30 21:41:21.176427 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:21.176402 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39fe3952-ddaf-4858-af83-458d1c01f0e8-istio-envoy\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m\" (UID: \"39fe3952-ddaf-4858-af83-458d1c01f0e8\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m" Oct 30 21:41:21.176493 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:21.176463 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39fe3952-ddaf-4858-af83-458d1c01f0e8-workload-certs\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m\" (UID: \"39fe3952-ddaf-4858-af83-458d1c01f0e8\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m" Oct 30 21:41:21.176602 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:21.176589 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39fe3952-ddaf-4858-af83-458d1c01f0e8-istiod-ca-cert\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m\" (UID: \"39fe3952-ddaf-4858-af83-458d1c01f0e8\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m" Oct 30 21:41:21.176662 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:21.176651 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39fe3952-ddaf-4858-af83-458d1c01f0e8-istio-podinfo\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m\" (UID: \"39fe3952-ddaf-4858-af83-458d1c01f0e8\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m" Oct 30 21:41:21.278366 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:21.278332 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39fe3952-ddaf-4858-af83-458d1c01f0e8-istiod-ca-cert\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m\" (UID: \"39fe3952-ddaf-4858-af83-458d1c01f0e8\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m" Oct 30 21:41:21.278580 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:21.278389 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39fe3952-ddaf-4858-af83-458d1c01f0e8-istio-podinfo\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m\" (UID: \"39fe3952-ddaf-4858-af83-458d1c01f0e8\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m" Oct 30 21:41:21.278580 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:21.278424 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39fe3952-ddaf-4858-af83-458d1c01f0e8-workload-socket\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m\" (UID: \"39fe3952-ddaf-4858-af83-458d1c01f0e8\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m" Oct 30 21:41:21.278580 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:21.278451 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39fe3952-ddaf-4858-af83-458d1c01f0e8-credential-socket\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m\" (UID: \"39fe3952-ddaf-4858-af83-458d1c01f0e8\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m" Oct 30 21:41:21.278580 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:21.278468 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqc2m\" (UniqueName: \"kubernetes.io/projected/39fe3952-ddaf-4858-af83-458d1c01f0e8-kube-api-access-tqc2m\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m\" (UID: \"39fe3952-ddaf-4858-af83-458d1c01f0e8\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m" Oct 30 21:41:21.278580 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:21.278537 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39fe3952-ddaf-4858-af83-458d1c01f0e8-istio-token\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m\" (UID: \"39fe3952-ddaf-4858-af83-458d1c01f0e8\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m" Oct 30 21:41:21.278580 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:21.278557 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39fe3952-ddaf-4858-af83-458d1c01f0e8-istio-data\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m\" (UID: \"39fe3952-ddaf-4858-af83-458d1c01f0e8\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m" Oct 30 21:41:21.278931 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:21.278593 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39fe3952-ddaf-4858-af83-458d1c01f0e8-istio-envoy\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m\" (UID: \"39fe3952-ddaf-4858-af83-458d1c01f0e8\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m" Oct 30 21:41:21.278931 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:21.278637 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39fe3952-ddaf-4858-af83-458d1c01f0e8-workload-certs\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m\" (UID: \"39fe3952-ddaf-4858-af83-458d1c01f0e8\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m" Oct 30 21:41:21.279049 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:21.278922 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39fe3952-ddaf-4858-af83-458d1c01f0e8-credential-socket\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m\" (UID: \"39fe3952-ddaf-4858-af83-458d1c01f0e8\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m" Oct 30 21:41:21.279049 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:21.278975 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39fe3952-ddaf-4858-af83-458d1c01f0e8-istio-data\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m\" (UID: \"39fe3952-ddaf-4858-af83-458d1c01f0e8\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m" Oct 30 21:41:21.279049 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:21.279020 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39fe3952-ddaf-4858-af83-458d1c01f0e8-workload-certs\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m\" (UID: \"39fe3952-ddaf-4858-af83-458d1c01f0e8\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m" Oct 30 21:41:21.279234 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:21.279207 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39fe3952-ddaf-4858-af83-458d1c01f0e8-istiod-ca-cert\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m\" (UID: \"39fe3952-ddaf-4858-af83-458d1c01f0e8\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m" Oct 30 21:41:21.279338 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:21.279315 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39fe3952-ddaf-4858-af83-458d1c01f0e8-workload-socket\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m\" (UID: \"39fe3952-ddaf-4858-af83-458d1c01f0e8\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m" Oct 30 21:41:21.280935 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:21.280913 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39fe3952-ddaf-4858-af83-458d1c01f0e8-istio-envoy\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m\" (UID: \"39fe3952-ddaf-4858-af83-458d1c01f0e8\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m" Oct 30 21:41:21.281039 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:21.281003 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39fe3952-ddaf-4858-af83-458d1c01f0e8-istio-podinfo\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m\" (UID: \"39fe3952-ddaf-4858-af83-458d1c01f0e8\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m" Oct 30 21:41:21.340609 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:21.340527 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39fe3952-ddaf-4858-af83-458d1c01f0e8-istio-token\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m\" (UID: \"39fe3952-ddaf-4858-af83-458d1c01f0e8\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m" Oct 30 21:41:21.340609 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:21.340581 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqc2m\" (UniqueName: \"kubernetes.io/projected/39fe3952-ddaf-4858-af83-458d1c01f0e8-kube-api-access-tqc2m\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m\" (UID: \"39fe3952-ddaf-4858-af83-458d1c01f0e8\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m" Oct 30 21:41:21.384226 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:21.384192 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m" Oct 30 21:41:21.821474 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:21.821451 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m"] Oct 30 21:41:21.822151 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:41:21.822126 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39fe3952_ddaf_4858_af83_458d1c01f0e8.slice/crio-74c73d0a805e64b79599f0262ead880992de351e0aea7a5b3566d355e1d5dbcf WatchSource:0}: Error finding container 74c73d0a805e64b79599f0262ead880992de351e0aea7a5b3566d355e1d5dbcf: Status 404 returned error can't find the container with id 74c73d0a805e64b79599f0262ead880992de351e0aea7a5b3566d355e1d5dbcf Oct 30 21:41:21.824587 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:21.824550 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:41:21.824708 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:21.824637 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:41:21.824708 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:21.824698 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:41:21.986316 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:21.986276 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m" event={"ID":"39fe3952-ddaf-4858-af83-458d1c01f0e8","Type":"ContainerStarted","Data":"b3989391e8071317375a147526b7af75d2101b4587f0de7ec89098f2d587b0bf"} Oct 30 21:41:21.986516 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:21.986346 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m" event={"ID":"39fe3952-ddaf-4858-af83-458d1c01f0e8","Type":"ContainerStarted","Data":"74c73d0a805e64b79599f0262ead880992de351e0aea7a5b3566d355e1d5dbcf"} Oct 30 21:41:22.110845 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:22.110744 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m" podStartSLOduration=2.11072957 podStartE2EDuration="2.11072957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:22.107747233 +0000 UTC m=+1065.850268168" watchObservedRunningTime="2026-10-30 21:41:22.11072957 +0000 UTC m=+1065.853250504" Oct 30 21:41:22.385139 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:22.385054 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m" Oct 30 21:41:22.386637 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:22.386616 5748 patch_prober.go:29] interesting pod/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 21:41:22.386702 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:22.386661 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m" podUID="39fe3952-ddaf-4858-af83-458d1c01f0e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 21:41:23.385398 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:23.385360 5748 patch_prober.go:29] interesting pod/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 21:41:23.385802 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:23.385413 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m" podUID="39fe3952-ddaf-4858-af83-458d1c01f0e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 21:41:24.385178 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:24.385136 5748 patch_prober.go:29] interesting pod/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 21:41:24.385366 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:24.385201 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m" podUID="39fe3952-ddaf-4858-af83-458d1c01f0e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 21:41:25.384669 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:25.384630 5748 patch_prober.go:29] interesting pod/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 21:41:25.385080 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:25.384694 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m" podUID="39fe3952-ddaf-4858-af83-458d1c01f0e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 21:41:26.384739 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:26.384702 5748 patch_prober.go:29] interesting pod/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 21:41:26.385199 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:26.384767 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m" podUID="39fe3952-ddaf-4858-af83-458d1c01f0e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 21:41:27.052002 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:27.051965 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-91874e/distinct-w2-91874e-kserve-58d6bdfc75-bdp5z" Oct 30 21:41:27.064534 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:27.064504 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-91874e/distinct-w2-91874e-kserve-58d6bdfc75-bdp5z" Oct 30 21:41:27.384879 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:27.384799 5748 patch_prober.go:29] interesting pod/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 21:41:27.384879 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:27.384854 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m" podUID="39fe3952-ddaf-4858-af83-458d1c01f0e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 21:41:28.060718 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:28.060683 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-91874e/distinct-w2-91874e-kserve-58d6bdfc75-bdp5z" Oct 30 21:41:28.384845 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:28.384764 5748 patch_prober.go:29] interesting pod/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 21:41:28.384845 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:28.384831 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m" podUID="39fe3952-ddaf-4858-af83-458d1c01f0e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 21:41:29.388828 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:29.388797 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m" Oct 30 21:41:29.389234 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:29.389105 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m" Oct 30 21:41:29.389854 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:41:29.389834 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m" Oct 30 21:41:30.145168 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:41:30.145130 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:42:00.145172 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:42:00.145130 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:42:00.425297 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:00.425206 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t"] Oct 30 21:42:00.429661 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:00.429635 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t" Oct 30 21:42:00.547327 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:00.547290 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e94f771-d6e2-4901-a055-04a4e905d8b8-credential-socket\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t\" (UID: \"5e94f771-d6e2-4901-a055-04a4e905d8b8\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t" Oct 30 21:42:00.547520 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:00.547343 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e94f771-d6e2-4901-a055-04a4e905d8b8-istio-token\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t\" (UID: \"5e94f771-d6e2-4901-a055-04a4e905d8b8\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t" Oct 30 21:42:00.547520 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:00.547378 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e94f771-d6e2-4901-a055-04a4e905d8b8-istiod-ca-cert\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t\" (UID: \"5e94f771-d6e2-4901-a055-04a4e905d8b8\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t" Oct 30 21:42:00.547520 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:00.547451 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e94f771-d6e2-4901-a055-04a4e905d8b8-istio-data\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t\" (UID: \"5e94f771-d6e2-4901-a055-04a4e905d8b8\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t" Oct 30 21:42:00.547520 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:00.547488 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e94f771-d6e2-4901-a055-04a4e905d8b8-workload-socket\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t\" (UID: \"5e94f771-d6e2-4901-a055-04a4e905d8b8\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t" Oct 30 21:42:00.547520 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:00.547504 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e94f771-d6e2-4901-a055-04a4e905d8b8-workload-certs\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t\" (UID: \"5e94f771-d6e2-4901-a055-04a4e905d8b8\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t" Oct 30 21:42:00.547776 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:00.547548 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e94f771-d6e2-4901-a055-04a4e905d8b8-istio-podinfo\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t\" (UID: \"5e94f771-d6e2-4901-a055-04a4e905d8b8\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t" Oct 30 21:42:00.547776 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:00.547589 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bq9q\" (UniqueName: \"kubernetes.io/projected/5e94f771-d6e2-4901-a055-04a4e905d8b8-kube-api-access-2bq9q\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t\" (UID: \"5e94f771-d6e2-4901-a055-04a4e905d8b8\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t" Oct 30 21:42:00.547776 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:00.547616 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e94f771-d6e2-4901-a055-04a4e905d8b8-istio-envoy\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t\" (UID: \"5e94f771-d6e2-4901-a055-04a4e905d8b8\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t" Oct 30 21:42:00.550439 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:00.550405 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t"] Oct 30 21:42:00.648728 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:00.648685 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e94f771-d6e2-4901-a055-04a4e905d8b8-credential-socket\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t\" (UID: \"5e94f771-d6e2-4901-a055-04a4e905d8b8\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t" Oct 30 21:42:00.648728 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:00.648737 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e94f771-d6e2-4901-a055-04a4e905d8b8-istio-token\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t\" (UID: \"5e94f771-d6e2-4901-a055-04a4e905d8b8\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t" Oct 30 21:42:00.649015 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:00.648757 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e94f771-d6e2-4901-a055-04a4e905d8b8-istiod-ca-cert\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t\" (UID: \"5e94f771-d6e2-4901-a055-04a4e905d8b8\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t" Oct 30 21:42:00.649015 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:00.648916 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e94f771-d6e2-4901-a055-04a4e905d8b8-istio-data\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t\" (UID: \"5e94f771-d6e2-4901-a055-04a4e905d8b8\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t" Oct 30 21:42:00.649015 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:00.648975 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e94f771-d6e2-4901-a055-04a4e905d8b8-workload-socket\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t\" (UID: \"5e94f771-d6e2-4901-a055-04a4e905d8b8\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t" Oct 30 21:42:00.649015 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:00.649002 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e94f771-d6e2-4901-a055-04a4e905d8b8-workload-certs\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t\" (UID: \"5e94f771-d6e2-4901-a055-04a4e905d8b8\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t" Oct 30 21:42:00.649246 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:00.649058 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e94f771-d6e2-4901-a055-04a4e905d8b8-istio-podinfo\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t\" (UID: \"5e94f771-d6e2-4901-a055-04a4e905d8b8\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t" Oct 30 21:42:00.649246 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:00.649129 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bq9q\" (UniqueName: \"kubernetes.io/projected/5e94f771-d6e2-4901-a055-04a4e905d8b8-kube-api-access-2bq9q\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t\" (UID: \"5e94f771-d6e2-4901-a055-04a4e905d8b8\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t" Oct 30 21:42:00.649246 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:00.649161 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e94f771-d6e2-4901-a055-04a4e905d8b8-istio-envoy\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t\" (UID: \"5e94f771-d6e2-4901-a055-04a4e905d8b8\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t" Oct 30 21:42:00.649408 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:00.649254 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e94f771-d6e2-4901-a055-04a4e905d8b8-credential-socket\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t\" (UID: \"5e94f771-d6e2-4901-a055-04a4e905d8b8\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t" Oct 30 21:42:00.649468 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:00.649402 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e94f771-d6e2-4901-a055-04a4e905d8b8-workload-certs\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t\" (UID: \"5e94f771-d6e2-4901-a055-04a4e905d8b8\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t" Oct 30 21:42:00.649468 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:00.649431 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e94f771-d6e2-4901-a055-04a4e905d8b8-istio-data\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t\" (UID: \"5e94f771-d6e2-4901-a055-04a4e905d8b8\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t" Oct 30 21:42:00.649577 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:00.649511 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e94f771-d6e2-4901-a055-04a4e905d8b8-workload-socket\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t\" (UID: \"5e94f771-d6e2-4901-a055-04a4e905d8b8\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t" Oct 30 21:42:00.650021 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:00.649996 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e94f771-d6e2-4901-a055-04a4e905d8b8-istiod-ca-cert\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t\" (UID: \"5e94f771-d6e2-4901-a055-04a4e905d8b8\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t" Oct 30 21:42:00.651512 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:00.651484 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e94f771-d6e2-4901-a055-04a4e905d8b8-istio-envoy\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t\" (UID: \"5e94f771-d6e2-4901-a055-04a4e905d8b8\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t" Oct 30 21:42:00.651676 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:00.651656 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e94f771-d6e2-4901-a055-04a4e905d8b8-istio-podinfo\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t\" (UID: \"5e94f771-d6e2-4901-a055-04a4e905d8b8\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t" Oct 30 21:42:00.710414 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:00.710359 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e94f771-d6e2-4901-a055-04a4e905d8b8-istio-token\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t\" (UID: \"5e94f771-d6e2-4901-a055-04a4e905d8b8\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t" Oct 30 21:42:00.710414 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:00.710380 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bq9q\" (UniqueName: \"kubernetes.io/projected/5e94f771-d6e2-4901-a055-04a4e905d8b8-kube-api-access-2bq9q\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t\" (UID: \"5e94f771-d6e2-4901-a055-04a4e905d8b8\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t" Oct 30 21:42:00.744150 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:00.744125 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t" Oct 30 21:42:01.184165 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:01.182833 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t"] Oct 30 21:42:01.184972 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:42:01.184938 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e94f771_d6e2_4901_a055_04a4e905d8b8.slice/crio-eca7bf3e66857343e72846f74d6d72cf62c3740e34fb6d7ac944eb2f684044b4 WatchSource:0}: Error finding container eca7bf3e66857343e72846f74d6d72cf62c3740e34fb6d7ac944eb2f684044b4: Status 404 returned error can't find the container with id eca7bf3e66857343e72846f74d6d72cf62c3740e34fb6d7ac944eb2f684044b4 Oct 30 21:42:01.187785 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:01.187750 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:42:01.187892 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:01.187820 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:42:01.187892 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:01.187848 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:42:01.283381 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:01.283329 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t" event={"ID":"5e94f771-d6e2-4901-a055-04a4e905d8b8","Type":"ContainerStarted","Data":"cf907fad51e3e8cbf09b907bc79c3f4d3dd665bd3ef4c96744a153bb3c9d47c6"} Oct 30 21:42:01.283536 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:01.283394 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t" event={"ID":"5e94f771-d6e2-4901-a055-04a4e905d8b8","Type":"ContainerStarted","Data":"eca7bf3e66857343e72846f74d6d72cf62c3740e34fb6d7ac944eb2f684044b4"} Oct 30 21:42:01.406551 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:01.406498 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t" podStartSLOduration=1.406484249 podStartE2EDuration="1.406484249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:01.40586759 +0000 UTC m=+1105.148388524" watchObservedRunningTime="2026-10-30 21:42:01.406484249 +0000 UTC m=+1105.149005214" Oct 30 21:42:01.744971 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:01.744923 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t" Oct 30 21:42:02.746612 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:02.746578 5748 patch_prober.go:29] interesting pod/e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 21:42:02.747020 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:02.746631 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t" podUID="5e94f771-d6e2-4901-a055-04a4e905d8b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 21:42:03.745308 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:03.745272 5748 patch_prober.go:29] interesting pod/e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 21:42:03.745475 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:03.745345 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t" podUID="5e94f771-d6e2-4901-a055-04a4e905d8b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 21:42:04.745445 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:04.745405 5748 patch_prober.go:29] interesting pod/e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 21:42:04.745848 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:04.745461 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t" podUID="5e94f771-d6e2-4901-a055-04a4e905d8b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 21:42:05.745255 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:05.745220 5748 patch_prober.go:29] interesting pod/e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 21:42:05.745444 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:05.745272 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t" podUID="5e94f771-d6e2-4901-a055-04a4e905d8b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 21:42:06.745449 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:06.745413 5748 patch_prober.go:29] interesting pod/e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 21:42:06.745856 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:06.745465 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t" podUID="5e94f771-d6e2-4901-a055-04a4e905d8b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 21:42:07.744375 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:07.744340 5748 patch_prober.go:29] interesting pod/e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 21:42:07.744584 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:07.744391 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t" podUID="5e94f771-d6e2-4901-a055-04a4e905d8b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 21:42:08.748422 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:08.748388 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t" Oct 30 21:42:08.748802 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:08.748665 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t" Oct 30 21:42:08.749301 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:08.749280 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t" Oct 30 21:42:21.142265 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:21.142220 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz"] Oct 30 21:42:21.146372 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:21.146347 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz" Oct 30 21:42:21.200260 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:21.200224 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-69e4b2-openshift-default-dockercfg-qvwv2\"" Oct 30 21:42:21.219030 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:21.218981 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b659bb43-102b-4936-87ed-d1b955262349-workload-certs\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz\" (UID: \"b659bb43-102b-4936-87ed-d1b955262349\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz" Oct 30 21:42:21.219030 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:21.219032 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b659bb43-102b-4936-87ed-d1b955262349-istio-podinfo\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz\" (UID: \"b659bb43-102b-4936-87ed-d1b955262349\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz" Oct 30 21:42:21.219252 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:21.219146 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b659bb43-102b-4936-87ed-d1b955262349-istio-envoy\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz\" (UID: \"b659bb43-102b-4936-87ed-d1b955262349\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz" Oct 30 21:42:21.219252 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:21.219205 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b659bb43-102b-4936-87ed-d1b955262349-istio-data\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz\" (UID: \"b659bb43-102b-4936-87ed-d1b955262349\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz" Oct 30 21:42:21.219321 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:21.219264 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b659bb43-102b-4936-87ed-d1b955262349-istio-token\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz\" (UID: \"b659bb43-102b-4936-87ed-d1b955262349\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz" Oct 30 21:42:21.219321 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:21.219314 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b659bb43-102b-4936-87ed-d1b955262349-credential-socket\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz\" (UID: \"b659bb43-102b-4936-87ed-d1b955262349\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz" Oct 30 21:42:21.219386 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:21.219354 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b659bb43-102b-4936-87ed-d1b955262349-workload-socket\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz\" (UID: \"b659bb43-102b-4936-87ed-d1b955262349\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz" Oct 30 21:42:21.219386 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:21.219375 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b659bb43-102b-4936-87ed-d1b955262349-istiod-ca-cert\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz\" (UID: \"b659bb43-102b-4936-87ed-d1b955262349\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz" Oct 30 21:42:21.219503 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:21.219485 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzqwk\" (UniqueName: \"kubernetes.io/projected/b659bb43-102b-4936-87ed-d1b955262349-kube-api-access-tzqwk\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz\" (UID: \"b659bb43-102b-4936-87ed-d1b955262349\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz" Oct 30 21:42:21.254587 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:21.254556 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz"] Oct 30 21:42:21.320934 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:21.320878 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b659bb43-102b-4936-87ed-d1b955262349-istio-data\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz\" (UID: \"b659bb43-102b-4936-87ed-d1b955262349\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz" Oct 30 21:42:21.321114 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:21.320945 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b659bb43-102b-4936-87ed-d1b955262349-istio-token\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz\" (UID: \"b659bb43-102b-4936-87ed-d1b955262349\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz" Oct 30 21:42:21.321114 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:21.320995 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b659bb43-102b-4936-87ed-d1b955262349-credential-socket\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz\" (UID: \"b659bb43-102b-4936-87ed-d1b955262349\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz" Oct 30 21:42:21.321114 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:21.321040 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b659bb43-102b-4936-87ed-d1b955262349-workload-socket\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz\" (UID: \"b659bb43-102b-4936-87ed-d1b955262349\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz" Oct 30 21:42:21.321295 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:21.321198 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b659bb43-102b-4936-87ed-d1b955262349-istiod-ca-cert\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz\" (UID: \"b659bb43-102b-4936-87ed-d1b955262349\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz" Oct 30 21:42:21.321295 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:21.321286 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzqwk\" (UniqueName: \"kubernetes.io/projected/b659bb43-102b-4936-87ed-d1b955262349-kube-api-access-tzqwk\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz\" (UID: \"b659bb43-102b-4936-87ed-d1b955262349\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz" Oct 30 21:42:21.321392 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:21.321330 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b659bb43-102b-4936-87ed-d1b955262349-istio-data\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz\" (UID: \"b659bb43-102b-4936-87ed-d1b955262349\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz" Oct 30 21:42:21.321392 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:21.321344 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b659bb43-102b-4936-87ed-d1b955262349-workload-socket\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz\" (UID: \"b659bb43-102b-4936-87ed-d1b955262349\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz" Oct 30 21:42:21.321466 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:21.321411 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b659bb43-102b-4936-87ed-d1b955262349-credential-socket\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz\" (UID: \"b659bb43-102b-4936-87ed-d1b955262349\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz" Oct 30 21:42:21.321466 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:21.321450 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b659bb43-102b-4936-87ed-d1b955262349-workload-certs\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz\" (UID: \"b659bb43-102b-4936-87ed-d1b955262349\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz" Oct 30 21:42:21.321546 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:21.321475 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b659bb43-102b-4936-87ed-d1b955262349-istio-podinfo\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz\" (UID: \"b659bb43-102b-4936-87ed-d1b955262349\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz" Oct 30 21:42:21.321546 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:21.321510 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b659bb43-102b-4936-87ed-d1b955262349-istio-envoy\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz\" (UID: \"b659bb43-102b-4936-87ed-d1b955262349\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz" Oct 30 21:42:21.321806 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:21.321784 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b659bb43-102b-4936-87ed-d1b955262349-workload-certs\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz\" (UID: \"b659bb43-102b-4936-87ed-d1b955262349\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz" Oct 30 21:42:21.321844 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:21.321811 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b659bb43-102b-4936-87ed-d1b955262349-istiod-ca-cert\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz\" (UID: \"b659bb43-102b-4936-87ed-d1b955262349\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz" Oct 30 21:42:21.323770 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:21.323747 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b659bb43-102b-4936-87ed-d1b955262349-istio-envoy\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz\" (UID: \"b659bb43-102b-4936-87ed-d1b955262349\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz" Oct 30 21:42:21.323994 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:21.323979 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b659bb43-102b-4936-87ed-d1b955262349-istio-podinfo\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz\" (UID: \"b659bb43-102b-4936-87ed-d1b955262349\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz" Oct 30 21:42:21.384836 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:21.384807 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b659bb43-102b-4936-87ed-d1b955262349-istio-token\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz\" (UID: \"b659bb43-102b-4936-87ed-d1b955262349\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz" Oct 30 21:42:21.384995 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:21.384980 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzqwk\" (UniqueName: \"kubernetes.io/projected/b659bb43-102b-4936-87ed-d1b955262349-kube-api-access-tzqwk\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz\" (UID: \"b659bb43-102b-4936-87ed-d1b955262349\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz" Oct 30 21:42:21.460693 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:21.460612 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz" Oct 30 21:42:21.893998 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:21.893972 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz"] Oct 30 21:42:21.895249 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:42:21.895215 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb659bb43_102b_4936_87ed_d1b955262349.slice/crio-c66447f107a9863602b7b7984266d253f9ad6498663d753c059a2670cd62b10d WatchSource:0}: Error finding container c66447f107a9863602b7b7984266d253f9ad6498663d753c059a2670cd62b10d: Status 404 returned error can't find the container with id c66447f107a9863602b7b7984266d253f9ad6498663d753c059a2670cd62b10d Oct 30 21:42:21.897715 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:21.897684 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:42:21.897803 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:21.897747 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:42:21.897803 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:21.897779 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:42:22.449908 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:22.449857 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz" event={"ID":"b659bb43-102b-4936-87ed-d1b955262349","Type":"ContainerStarted","Data":"72169f1df2df2264c2602a3e930e762511bb1882bfa9eed5e62397e130283724"} Oct 30 21:42:22.450363 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:22.449933 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz" event={"ID":"b659bb43-102b-4936-87ed-d1b955262349","Type":"ContainerStarted","Data":"c66447f107a9863602b7b7984266d253f9ad6498663d753c059a2670cd62b10d"} Oct 30 21:42:22.461200 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:22.461172 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz" Oct 30 21:42:22.463010 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:22.462986 5748 patch_prober.go:29] interesting pod/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 21:42:22.463119 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:22.463033 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz" podUID="b659bb43-102b-4936-87ed-d1b955262349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 21:42:22.580627 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:22.580574 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz" podStartSLOduration=1.580561128 podStartE2EDuration="1.580561128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:22.57729953 +0000 UTC m=+1126.319820467" watchObservedRunningTime="2026-10-30 21:42:22.580561128 +0000 UTC m=+1126.323082140" Oct 30 21:42:23.461271 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:23.461240 5748 patch_prober.go:29] interesting pod/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 21:42:23.461662 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:23.461290 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz" podUID="b659bb43-102b-4936-87ed-d1b955262349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 21:42:24.461117 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:24.461082 5748 patch_prober.go:29] interesting pod/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 21:42:24.461319 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:24.461143 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz" podUID="b659bb43-102b-4936-87ed-d1b955262349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 21:42:25.461124 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:25.461087 5748 patch_prober.go:29] interesting pod/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 21:42:25.461422 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:25.461145 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz" podUID="b659bb43-102b-4936-87ed-d1b955262349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 21:42:26.463290 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:26.463251 5748 patch_prober.go:29] interesting pod/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 21:42:26.465731 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:26.463786 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz" podUID="b659bb43-102b-4936-87ed-d1b955262349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 21:42:27.461073 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:27.461030 5748 patch_prober.go:29] interesting pod/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 21:42:27.461258 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:27.461090 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz" podUID="b659bb43-102b-4936-87ed-d1b955262349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 21:42:28.461730 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:28.461695 5748 patch_prober.go:29] interesting pod/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 21:42:28.462243 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:28.461754 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz" podUID="b659bb43-102b-4936-87ed-d1b955262349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 21:42:29.465385 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:29.465352 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz" Oct 30 21:42:29.465937 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:29.465755 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz" Oct 30 21:42:29.466550 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:29.466527 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz" Oct 30 21:42:30.147824 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:42:30.147778 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:42:55.627798 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:55.627759 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92937c3f-nwdx8"] Oct 30 21:42:55.632071 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:55.632051 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92937c3f-nwdx8" Oct 30 21:42:55.737707 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:55.737668 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:42:55.737707 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:55.737695 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:42:55.737928 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:55.737756 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:42:55.737928 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:55.737830 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 21:42:55.741273 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:55.741249 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92937c3f-nwdx8"] Oct 30 21:42:55.756317 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:55.756295 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tjkr\" (UniqueName: \"kubernetes.io/projected/f5171ada-77cf-4fc9-95b9-040f31458098-kube-api-access-5tjkr\") pod \"maas-api-revoke-sub-92937c3f-nwdx8\" (UID: \"f5171ada-77cf-4fc9-95b9-040f31458098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92937c3f-nwdx8" Oct 30 21:42:55.756464 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:55.756380 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5171ada-77cf-4fc9-95b9-040f31458098-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92937c3f-nwdx8\" (UID: \"f5171ada-77cf-4fc9-95b9-040f31458098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92937c3f-nwdx8" Oct 30 21:42:55.756507 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:55.756465 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5171ada-77cf-4fc9-95b9-040f31458098-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92937c3f-nwdx8\" (UID: \"f5171ada-77cf-4fc9-95b9-040f31458098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92937c3f-nwdx8" Oct 30 21:42:55.857559 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:55.857526 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5171ada-77cf-4fc9-95b9-040f31458098-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92937c3f-nwdx8\" (UID: \"f5171ada-77cf-4fc9-95b9-040f31458098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92937c3f-nwdx8" Oct 30 21:42:55.857745 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:55.857591 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5171ada-77cf-4fc9-95b9-040f31458098-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92937c3f-nwdx8\" (UID: \"f5171ada-77cf-4fc9-95b9-040f31458098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92937c3f-nwdx8" Oct 30 21:42:55.857745 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:55.857669 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tjkr\" (UniqueName: \"kubernetes.io/projected/f5171ada-77cf-4fc9-95b9-040f31458098-kube-api-access-5tjkr\") pod \"maas-api-revoke-sub-92937c3f-nwdx8\" (UID: \"f5171ada-77cf-4fc9-95b9-040f31458098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92937c3f-nwdx8" Oct 30 21:42:55.858307 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:55.858280 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5171ada-77cf-4fc9-95b9-040f31458098-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92937c3f-nwdx8\" (UID: \"f5171ada-77cf-4fc9-95b9-040f31458098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92937c3f-nwdx8" Oct 30 21:42:55.859983 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:55.859956 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5171ada-77cf-4fc9-95b9-040f31458098-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92937c3f-nwdx8\" (UID: \"f5171ada-77cf-4fc9-95b9-040f31458098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92937c3f-nwdx8" Oct 30 21:42:55.924513 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:55.924481 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tjkr\" (UniqueName: \"kubernetes.io/projected/f5171ada-77cf-4fc9-95b9-040f31458098-kube-api-access-5tjkr\") pod \"maas-api-revoke-sub-92937c3f-nwdx8\" (UID: \"f5171ada-77cf-4fc9-95b9-040f31458098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92937c3f-nwdx8" Oct 30 21:42:55.947353 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:55.947324 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92937c3f-nwdx8" Oct 30 21:42:56.338659 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:56.338628 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92937c3f-nwdx8"] Oct 30 21:42:56.338867 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:42:56.338829 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5171ada_77cf_4fc9_95b9_040f31458098.slice/crio-af0fcdef41dc0b31cefc030c91b9b5ef40de9dfe3e93ee94f3b455bbef89910b WatchSource:0}: Error finding container af0fcdef41dc0b31cefc030c91b9b5ef40de9dfe3e93ee94f3b455bbef89910b: Status 404 returned error can't find the container with id af0fcdef41dc0b31cefc030c91b9b5ef40de9dfe3e93ee94f3b455bbef89910b Oct 30 21:42:56.690622 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:56.690579 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92937c3f-nwdx8" event={"ID":"f5171ada-77cf-4fc9-95b9-040f31458098","Type":"ContainerStarted","Data":"1236039daad2d4cdc580ffc77b1c2ee82430557f25cf0c08030a4f889b7a16c4"} Oct 30 21:42:56.691082 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:56.690644 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92937c3f-nwdx8" event={"ID":"f5171ada-77cf-4fc9-95b9-040f31458098","Type":"ContainerStarted","Data":"af0fcdef41dc0b31cefc030c91b9b5ef40de9dfe3e93ee94f3b455bbef89910b"} Oct 30 21:42:56.824026 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:56.823974 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92937c3f-nwdx8" podStartSLOduration=1.823958397 podStartE2EDuration="1.823958397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:56.82212809 +0000 UTC m=+1160.564649029" watchObservedRunningTime="2026-10-30 21:42:56.823958397 +0000 UTC m=+1160.566479384" Oct 30 21:42:57.698349 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:57.698316 5748 generic.go:356] "Generic (PLEG): container finished" podID="f5171ada-77cf-4fc9-95b9-040f31458098" containerID="1236039daad2d4cdc580ffc77b1c2ee82430557f25cf0c08030a4f889b7a16c4" exitCode=0 Oct 30 21:42:57.698770 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:57.698388 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92937c3f-nwdx8" event={"ID":"f5171ada-77cf-4fc9-95b9-040f31458098","Type":"ContainerDied","Data":"1236039daad2d4cdc580ffc77b1c2ee82430557f25cf0c08030a4f889b7a16c4"} Oct 30 21:42:58.851527 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:58.851501 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92937c3f-nwdx8" Oct 30 21:42:58.995377 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:58.995305 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5171ada-77cf-4fc9-95b9-040f31458098-kube-api-access-5tjkr\" (UniqueName: \"kubernetes.io/projected/f5171ada-77cf-4fc9-95b9-040f31458098-kube-api-access-5tjkr\") pod \"f5171ada-77cf-4fc9-95b9-040f31458098\" (UID: \"f5171ada-77cf-4fc9-95b9-040f31458098\") " Oct 30 21:42:58.995547 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:58.995418 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5171ada-77cf-4fc9-95b9-040f31458098-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5171ada-77cf-4fc9-95b9-040f31458098-maas-api-openshift-ca\") pod \"f5171ada-77cf-4fc9-95b9-040f31458098\" (UID: \"f5171ada-77cf-4fc9-95b9-040f31458098\") " Oct 30 21:42:58.995547 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:58.995457 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5171ada-77cf-4fc9-95b9-040f31458098-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5171ada-77cf-4fc9-95b9-040f31458098-maas-api-xks-ca\") pod \"f5171ada-77cf-4fc9-95b9-040f31458098\" (UID: \"f5171ada-77cf-4fc9-95b9-040f31458098\") " Oct 30 21:42:58.995831 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:58.995801 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5171ada-77cf-4fc9-95b9-040f31458098-maas-api-openshift-ca" pod "f5171ada-77cf-4fc9-95b9-040f31458098" (UID: "f5171ada-77cf-4fc9-95b9-040f31458098"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:42:58.996006 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:58.995959 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5171ada-77cf-4fc9-95b9-040f31458098-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:58.997574 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:58.997546 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5171ada-77cf-4fc9-95b9-040f31458098-maas-api-xks-ca" pod "f5171ada-77cf-4fc9-95b9-040f31458098" (UID: "f5171ada-77cf-4fc9-95b9-040f31458098"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:42:58.997574 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:58.997566 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5171ada-77cf-4fc9-95b9-040f31458098-kube-api-access-5tjkr" pod "f5171ada-77cf-4fc9-95b9-040f31458098" (UID: "f5171ada-77cf-4fc9-95b9-040f31458098"). InnerVolumeSpecName "kube-api-access-5tjkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:42:59.097509 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:59.097474 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5171ada-77cf-4fc9-95b9-040f31458098-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:59.097509 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:59.097504 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tjkr\" (UniqueName: \"kubernetes.io/projected/f5171ada-77cf-4fc9-95b9-040f31458098-kube-api-access-5tjkr\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:59.712995 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:59.712963 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92937c3f-nwdx8" Oct 30 21:42:59.713177 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:59.713003 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92937c3f-nwdx8" event={"ID":"f5171ada-77cf-4fc9-95b9-040f31458098","Type":"ContainerDied","Data":"af0fcdef41dc0b31cefc030c91b9b5ef40de9dfe3e93ee94f3b455bbef89910b"} Oct 30 21:42:59.713177 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:42:59.713059 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af0fcdef41dc0b31cefc030c91b9b5ef40de9dfe3e93ee94f3b455bbef89910b" Oct 30 21:43:00.144864 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:43:00.144824 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:43:19.137445 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:19.137409 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-6ea374/distinct-2-w1-6ea374-kserve-68fb49889-hpg2k"] Oct 30 21:43:19.137968 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:19.137954 5748 cpu_manager.go:414] "containerMap: removing container" podUID="f5171ada-77cf-4fc9-95b9-040f31458098" containerName="revoke-keys" Oct 30 21:43:19.138019 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:19.137969 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5171ada-77cf-4fc9-95b9-040f31458098" containerName="revoke-keys" Oct 30 21:43:19.138098 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:19.138089 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5171ada-77cf-4fc9-95b9-040f31458098" containerName="revoke-keys" Oct 30 21:43:19.141404 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:19.141388 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6ea374/distinct-2-w1-6ea374-kserve-68fb49889-hpg2k" Oct 30 21:43:19.247501 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:19.247466 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab533734-e0d8-4acb-854d-12851a87bc58-tmp-dir\") pod \"distinct-2-w1-6ea374-kserve-68fb49889-hpg2k\" (UID: \"ab533734-e0d8-4acb-854d-12851a87bc58\") " pod="e2e-models-e2e-worker-w1-6ea374/distinct-2-w1-6ea374-kserve-68fb49889-hpg2k" Oct 30 21:43:19.247668 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:19.247628 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab533734-e0d8-4acb-854d-12851a87bc58-tls-certs\") pod \"distinct-2-w1-6ea374-kserve-68fb49889-hpg2k\" (UID: \"ab533734-e0d8-4acb-854d-12851a87bc58\") " pod="e2e-models-e2e-worker-w1-6ea374/distinct-2-w1-6ea374-kserve-68fb49889-hpg2k" Oct 30 21:43:19.247718 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:19.247676 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab533734-e0d8-4acb-854d-12851a87bc58-dshm\") pod \"distinct-2-w1-6ea374-kserve-68fb49889-hpg2k\" (UID: \"ab533734-e0d8-4acb-854d-12851a87bc58\") " pod="e2e-models-e2e-worker-w1-6ea374/distinct-2-w1-6ea374-kserve-68fb49889-hpg2k" Oct 30 21:43:19.247782 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:19.247759 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jdf8\" (UniqueName: \"kubernetes.io/projected/ab533734-e0d8-4acb-854d-12851a87bc58-kube-api-access-8jdf8\") pod \"distinct-2-w1-6ea374-kserve-68fb49889-hpg2k\" (UID: \"ab533734-e0d8-4acb-854d-12851a87bc58\") " pod="e2e-models-e2e-worker-w1-6ea374/distinct-2-w1-6ea374-kserve-68fb49889-hpg2k" Oct 30 21:43:19.247907 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:19.247874 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab533734-e0d8-4acb-854d-12851a87bc58-model-cache\") pod \"distinct-2-w1-6ea374-kserve-68fb49889-hpg2k\" (UID: \"ab533734-e0d8-4acb-854d-12851a87bc58\") " pod="e2e-models-e2e-worker-w1-6ea374/distinct-2-w1-6ea374-kserve-68fb49889-hpg2k" Oct 30 21:43:19.248016 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:19.247996 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab533734-e0d8-4acb-854d-12851a87bc58-home\") pod \"distinct-2-w1-6ea374-kserve-68fb49889-hpg2k\" (UID: \"ab533734-e0d8-4acb-854d-12851a87bc58\") " pod="e2e-models-e2e-worker-w1-6ea374/distinct-2-w1-6ea374-kserve-68fb49889-hpg2k" Oct 30 21:43:19.248151 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:19.248131 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-6ea374\"/\"distinct-2-w1-6ea374-kserve-self-signed-certs\"" Oct 30 21:43:19.251768 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:19.251749 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-6ea374/distinct-2-w1-6ea374-kserve-68fb49889-hpg2k"] Oct 30 21:43:19.350094 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:19.350056 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab533734-e0d8-4acb-854d-12851a87bc58-tls-certs\") pod \"distinct-2-w1-6ea374-kserve-68fb49889-hpg2k\" (UID: \"ab533734-e0d8-4acb-854d-12851a87bc58\") " pod="e2e-models-e2e-worker-w1-6ea374/distinct-2-w1-6ea374-kserve-68fb49889-hpg2k" Oct 30 21:43:19.350312 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:19.350119 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab533734-e0d8-4acb-854d-12851a87bc58-dshm\") pod \"distinct-2-w1-6ea374-kserve-68fb49889-hpg2k\" (UID: \"ab533734-e0d8-4acb-854d-12851a87bc58\") " pod="e2e-models-e2e-worker-w1-6ea374/distinct-2-w1-6ea374-kserve-68fb49889-hpg2k" Oct 30 21:43:19.350312 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:19.350150 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jdf8\" (UniqueName: \"kubernetes.io/projected/ab533734-e0d8-4acb-854d-12851a87bc58-kube-api-access-8jdf8\") pod \"distinct-2-w1-6ea374-kserve-68fb49889-hpg2k\" (UID: \"ab533734-e0d8-4acb-854d-12851a87bc58\") " pod="e2e-models-e2e-worker-w1-6ea374/distinct-2-w1-6ea374-kserve-68fb49889-hpg2k" Oct 30 21:43:19.350312 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:19.350226 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab533734-e0d8-4acb-854d-12851a87bc58-model-cache\") pod \"distinct-2-w1-6ea374-kserve-68fb49889-hpg2k\" (UID: \"ab533734-e0d8-4acb-854d-12851a87bc58\") " pod="e2e-models-e2e-worker-w1-6ea374/distinct-2-w1-6ea374-kserve-68fb49889-hpg2k" Oct 30 21:43:19.350474 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:19.350334 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab533734-e0d8-4acb-854d-12851a87bc58-home\") pod \"distinct-2-w1-6ea374-kserve-68fb49889-hpg2k\" (UID: \"ab533734-e0d8-4acb-854d-12851a87bc58\") " pod="e2e-models-e2e-worker-w1-6ea374/distinct-2-w1-6ea374-kserve-68fb49889-hpg2k" Oct 30 21:43:19.350474 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:19.350382 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab533734-e0d8-4acb-854d-12851a87bc58-tmp-dir\") pod \"distinct-2-w1-6ea374-kserve-68fb49889-hpg2k\" (UID: \"ab533734-e0d8-4acb-854d-12851a87bc58\") " pod="e2e-models-e2e-worker-w1-6ea374/distinct-2-w1-6ea374-kserve-68fb49889-hpg2k" Oct 30 21:43:19.351094 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:19.351058 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab533734-e0d8-4acb-854d-12851a87bc58-model-cache\") pod \"distinct-2-w1-6ea374-kserve-68fb49889-hpg2k\" (UID: \"ab533734-e0d8-4acb-854d-12851a87bc58\") " pod="e2e-models-e2e-worker-w1-6ea374/distinct-2-w1-6ea374-kserve-68fb49889-hpg2k" Oct 30 21:43:19.351343 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:19.351320 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab533734-e0d8-4acb-854d-12851a87bc58-home\") pod \"distinct-2-w1-6ea374-kserve-68fb49889-hpg2k\" (UID: \"ab533734-e0d8-4acb-854d-12851a87bc58\") " pod="e2e-models-e2e-worker-w1-6ea374/distinct-2-w1-6ea374-kserve-68fb49889-hpg2k" Oct 30 21:43:19.351722 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:19.351692 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab533734-e0d8-4acb-854d-12851a87bc58-tmp-dir\") pod \"distinct-2-w1-6ea374-kserve-68fb49889-hpg2k\" (UID: \"ab533734-e0d8-4acb-854d-12851a87bc58\") " pod="e2e-models-e2e-worker-w1-6ea374/distinct-2-w1-6ea374-kserve-68fb49889-hpg2k" Oct 30 21:43:19.353859 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:19.353836 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab533734-e0d8-4acb-854d-12851a87bc58-dshm\") pod \"distinct-2-w1-6ea374-kserve-68fb49889-hpg2k\" (UID: \"ab533734-e0d8-4acb-854d-12851a87bc58\") " pod="e2e-models-e2e-worker-w1-6ea374/distinct-2-w1-6ea374-kserve-68fb49889-hpg2k" Oct 30 21:43:19.354151 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:19.353847 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab533734-e0d8-4acb-854d-12851a87bc58-tls-certs\") pod \"distinct-2-w1-6ea374-kserve-68fb49889-hpg2k\" (UID: \"ab533734-e0d8-4acb-854d-12851a87bc58\") " pod="e2e-models-e2e-worker-w1-6ea374/distinct-2-w1-6ea374-kserve-68fb49889-hpg2k" Oct 30 21:43:19.412680 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:19.412611 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jdf8\" (UniqueName: \"kubernetes.io/projected/ab533734-e0d8-4acb-854d-12851a87bc58-kube-api-access-8jdf8\") pod \"distinct-2-w1-6ea374-kserve-68fb49889-hpg2k\" (UID: \"ab533734-e0d8-4acb-854d-12851a87bc58\") " pod="e2e-models-e2e-worker-w1-6ea374/distinct-2-w1-6ea374-kserve-68fb49889-hpg2k" Oct 30 21:43:19.454311 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:19.454276 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6ea374/distinct-2-w1-6ea374-kserve-68fb49889-hpg2k" Oct 30 21:43:19.854553 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:19.854515 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-6ea374/distinct-2-w1-6ea374-kserve-68fb49889-hpg2k"] Oct 30 21:43:19.856128 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:43:19.856104 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab533734_e0d8_4acb_854d_12851a87bc58.slice/crio-1ac38ee86ae98400b1272c3aa7869d630632381dd7b633ed43117d4ce80204ed WatchSource:0}: Error finding container 1ac38ee86ae98400b1272c3aa7869d630632381dd7b633ed43117d4ce80204ed: Status 404 returned error can't find the container with id 1ac38ee86ae98400b1272c3aa7869d630632381dd7b633ed43117d4ce80204ed Oct 30 21:43:20.856191 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:20.856157 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6ea374/distinct-2-w1-6ea374-kserve-68fb49889-hpg2k" event={"ID":"ab533734-e0d8-4acb-854d-12851a87bc58","Type":"ContainerStarted","Data":"74b28c950466bf3af4dedf9b1a99435428282a5233f0320db13e4042448e598f"} Oct 30 21:43:20.856191 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:20.856198 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6ea374/distinct-2-w1-6ea374-kserve-68fb49889-hpg2k" event={"ID":"ab533734-e0d8-4acb-854d-12851a87bc58","Type":"ContainerStarted","Data":"1ac38ee86ae98400b1272c3aa7869d630632381dd7b633ed43117d4ce80204ed"} Oct 30 21:43:20.978622 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:20.978564 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-6ea374/distinct-2-w1-6ea374-kserve-68fb49889-hpg2k" podStartSLOduration=1.9785504779999998 podStartE2EDuration="1.978550478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:20.976069847 +0000 UTC m=+1184.718590807" watchObservedRunningTime="2026-10-30 21:43:20.978550478 +0000 UTC m=+1184.721071425" Oct 30 21:43:21.455275 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:21.455244 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6ea374/distinct-2-w1-6ea374-kserve-68fb49889-hpg2k" Oct 30 21:43:29.454684 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:29.454645 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-6ea374/distinct-2-w1-6ea374-kserve-68fb49889-hpg2k" Oct 30 21:43:29.467382 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:29.467357 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-6ea374/distinct-2-w1-6ea374-kserve-68fb49889-hpg2k" Oct 30 21:43:29.639635 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:29.639599 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-91874e/distinct-2-w2-91874e-kserve-69fcf64f44-qm96r"] Oct 30 21:43:29.644013 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:29.643993 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-91874e/distinct-2-w2-91874e-kserve-69fcf64f44-qm96r" Oct 30 21:43:29.699975 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:29.699890 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-91874e\"/\"distinct-2-w2-91874e-kserve-self-signed-certs\"" Oct 30 21:43:29.705846 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:29.705758 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-91874e/distinct-2-w2-91874e-kserve-69fcf64f44-qm96r"] Oct 30 21:43:29.783780 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:29.783747 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cb16f33-b208-4c11-bb12-322829bd51e5-tls-certs\") pod \"distinct-2-w2-91874e-kserve-69fcf64f44-qm96r\" (UID: \"0cb16f33-b208-4c11-bb12-322829bd51e5\") " pod="e2e-models-e2e-worker-w2-91874e/distinct-2-w2-91874e-kserve-69fcf64f44-qm96r" Oct 30 21:43:29.783780 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:29.783782 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cb16f33-b208-4c11-bb12-322829bd51e5-model-cache\") pod \"distinct-2-w2-91874e-kserve-69fcf64f44-qm96r\" (UID: \"0cb16f33-b208-4c11-bb12-322829bd51e5\") " pod="e2e-models-e2e-worker-w2-91874e/distinct-2-w2-91874e-kserve-69fcf64f44-qm96r" Oct 30 21:43:29.784064 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:29.783936 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cb16f33-b208-4c11-bb12-322829bd51e5-home\") pod \"distinct-2-w2-91874e-kserve-69fcf64f44-qm96r\" (UID: \"0cb16f33-b208-4c11-bb12-322829bd51e5\") " pod="e2e-models-e2e-worker-w2-91874e/distinct-2-w2-91874e-kserve-69fcf64f44-qm96r" Oct 30 21:43:29.784064 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:29.783972 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cb16f33-b208-4c11-bb12-322829bd51e5-dshm\") pod \"distinct-2-w2-91874e-kserve-69fcf64f44-qm96r\" (UID: \"0cb16f33-b208-4c11-bb12-322829bd51e5\") " pod="e2e-models-e2e-worker-w2-91874e/distinct-2-w2-91874e-kserve-69fcf64f44-qm96r" Oct 30 21:43:29.784064 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:29.784010 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsnt2\" (UniqueName: \"kubernetes.io/projected/0cb16f33-b208-4c11-bb12-322829bd51e5-kube-api-access-bsnt2\") pod \"distinct-2-w2-91874e-kserve-69fcf64f44-qm96r\" (UID: \"0cb16f33-b208-4c11-bb12-322829bd51e5\") " pod="e2e-models-e2e-worker-w2-91874e/distinct-2-w2-91874e-kserve-69fcf64f44-qm96r" Oct 30 21:43:29.784230 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:29.784136 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cb16f33-b208-4c11-bb12-322829bd51e5-tmp-dir\") pod \"distinct-2-w2-91874e-kserve-69fcf64f44-qm96r\" (UID: \"0cb16f33-b208-4c11-bb12-322829bd51e5\") " pod="e2e-models-e2e-worker-w2-91874e/distinct-2-w2-91874e-kserve-69fcf64f44-qm96r" Oct 30 21:43:29.885510 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:29.885477 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cb16f33-b208-4c11-bb12-322829bd51e5-tmp-dir\") pod \"distinct-2-w2-91874e-kserve-69fcf64f44-qm96r\" (UID: \"0cb16f33-b208-4c11-bb12-322829bd51e5\") " pod="e2e-models-e2e-worker-w2-91874e/distinct-2-w2-91874e-kserve-69fcf64f44-qm96r" Oct 30 21:43:29.885708 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:29.885527 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cb16f33-b208-4c11-bb12-322829bd51e5-tls-certs\") pod \"distinct-2-w2-91874e-kserve-69fcf64f44-qm96r\" (UID: \"0cb16f33-b208-4c11-bb12-322829bd51e5\") " pod="e2e-models-e2e-worker-w2-91874e/distinct-2-w2-91874e-kserve-69fcf64f44-qm96r" Oct 30 21:43:29.885708 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:29.885660 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cb16f33-b208-4c11-bb12-322829bd51e5-model-cache\") pod \"distinct-2-w2-91874e-kserve-69fcf64f44-qm96r\" (UID: \"0cb16f33-b208-4c11-bb12-322829bd51e5\") " pod="e2e-models-e2e-worker-w2-91874e/distinct-2-w2-91874e-kserve-69fcf64f44-qm96r" Oct 30 21:43:29.885810 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:29.885783 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cb16f33-b208-4c11-bb12-322829bd51e5-home\") pod \"distinct-2-w2-91874e-kserve-69fcf64f44-qm96r\" (UID: \"0cb16f33-b208-4c11-bb12-322829bd51e5\") " pod="e2e-models-e2e-worker-w2-91874e/distinct-2-w2-91874e-kserve-69fcf64f44-qm96r" Oct 30 21:43:29.885857 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:29.885838 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cb16f33-b208-4c11-bb12-322829bd51e5-dshm\") pod \"distinct-2-w2-91874e-kserve-69fcf64f44-qm96r\" (UID: \"0cb16f33-b208-4c11-bb12-322829bd51e5\") " pod="e2e-models-e2e-worker-w2-91874e/distinct-2-w2-91874e-kserve-69fcf64f44-qm96r" Oct 30 21:43:29.885941 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:29.885923 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsnt2\" (UniqueName: \"kubernetes.io/projected/0cb16f33-b208-4c11-bb12-322829bd51e5-kube-api-access-bsnt2\") pod \"distinct-2-w2-91874e-kserve-69fcf64f44-qm96r\" (UID: \"0cb16f33-b208-4c11-bb12-322829bd51e5\") " pod="e2e-models-e2e-worker-w2-91874e/distinct-2-w2-91874e-kserve-69fcf64f44-qm96r" Oct 30 21:43:29.886013 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:29.885964 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cb16f33-b208-4c11-bb12-322829bd51e5-tmp-dir\") pod \"distinct-2-w2-91874e-kserve-69fcf64f44-qm96r\" (UID: \"0cb16f33-b208-4c11-bb12-322829bd51e5\") " pod="e2e-models-e2e-worker-w2-91874e/distinct-2-w2-91874e-kserve-69fcf64f44-qm96r" Oct 30 21:43:29.886074 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:29.886040 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cb16f33-b208-4c11-bb12-322829bd51e5-model-cache\") pod \"distinct-2-w2-91874e-kserve-69fcf64f44-qm96r\" (UID: \"0cb16f33-b208-4c11-bb12-322829bd51e5\") " pod="e2e-models-e2e-worker-w2-91874e/distinct-2-w2-91874e-kserve-69fcf64f44-qm96r" Oct 30 21:43:29.886414 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:29.886386 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cb16f33-b208-4c11-bb12-322829bd51e5-home\") pod \"distinct-2-w2-91874e-kserve-69fcf64f44-qm96r\" (UID: \"0cb16f33-b208-4c11-bb12-322829bd51e5\") " pod="e2e-models-e2e-worker-w2-91874e/distinct-2-w2-91874e-kserve-69fcf64f44-qm96r" Oct 30 21:43:29.888059 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:29.888038 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cb16f33-b208-4c11-bb12-322829bd51e5-dshm\") pod \"distinct-2-w2-91874e-kserve-69fcf64f44-qm96r\" (UID: \"0cb16f33-b208-4c11-bb12-322829bd51e5\") " pod="e2e-models-e2e-worker-w2-91874e/distinct-2-w2-91874e-kserve-69fcf64f44-qm96r" Oct 30 21:43:29.888270 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:29.888252 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cb16f33-b208-4c11-bb12-322829bd51e5-tls-certs\") pod \"distinct-2-w2-91874e-kserve-69fcf64f44-qm96r\" (UID: \"0cb16f33-b208-4c11-bb12-322829bd51e5\") " pod="e2e-models-e2e-worker-w2-91874e/distinct-2-w2-91874e-kserve-69fcf64f44-qm96r" Oct 30 21:43:29.931433 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:29.931405 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6ea374/distinct-2-w1-6ea374-kserve-68fb49889-hpg2k" Oct 30 21:43:29.947813 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:29.947779 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsnt2\" (UniqueName: \"kubernetes.io/projected/0cb16f33-b208-4c11-bb12-322829bd51e5-kube-api-access-bsnt2\") pod \"distinct-2-w2-91874e-kserve-69fcf64f44-qm96r\" (UID: \"0cb16f33-b208-4c11-bb12-322829bd51e5\") " pod="e2e-models-e2e-worker-w2-91874e/distinct-2-w2-91874e-kserve-69fcf64f44-qm96r" Oct 30 21:43:29.956510 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:29.956431 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-91874e/distinct-2-w2-91874e-kserve-69fcf64f44-qm96r" Oct 30 21:43:30.172712 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:43:30.172679 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:43:30.373590 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:30.373563 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-91874e/distinct-2-w2-91874e-kserve-69fcf64f44-qm96r"] Oct 30 21:43:30.374561 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:43:30.374539 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0cb16f33_b208_4c11_bb12_322829bd51e5.slice/crio-e3a19d7d1a0c64fc8b7481b10fbdd75284cf4a6b59dede991c44d470d0712eda WatchSource:0}: Error finding container e3a19d7d1a0c64fc8b7481b10fbdd75284cf4a6b59dede991c44d470d0712eda: Status 404 returned error can't find the container with id e3a19d7d1a0c64fc8b7481b10fbdd75284cf4a6b59dede991c44d470d0712eda Oct 30 21:43:30.929062 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:30.929021 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-91874e/distinct-2-w2-91874e-kserve-69fcf64f44-qm96r" event={"ID":"0cb16f33-b208-4c11-bb12-322829bd51e5","Type":"ContainerStarted","Data":"d9b311dbf4041d0a7755d6b6dddfbdd65a0c4bcb499249ea28327477f14ea58e"} Oct 30 21:43:30.929498 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:30.929073 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-91874e/distinct-2-w2-91874e-kserve-69fcf64f44-qm96r" event={"ID":"0cb16f33-b208-4c11-bb12-322829bd51e5","Type":"ContainerStarted","Data":"e3a19d7d1a0c64fc8b7481b10fbdd75284cf4a6b59dede991c44d470d0712eda"} Oct 30 21:43:30.957200 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:30.957164 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-91874e/distinct-2-w2-91874e-kserve-69fcf64f44-qm96r" Oct 30 21:43:31.051604 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:31.051543 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-91874e/distinct-2-w2-91874e-kserve-69fcf64f44-qm96r" podStartSLOduration=2.051525548 podStartE2EDuration="2.051525548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:31.050834377 +0000 UTC m=+1194.793355311" watchObservedRunningTime="2026-10-30 21:43:31.051525548 +0000 UTC m=+1194.794046484" Oct 30 21:43:31.103596 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:31.103557 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0857c785-bt6s8"] Oct 30 21:43:31.107004 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:31.106975 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0857c785-bt6s8"] Oct 30 21:43:32.417713 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:32.417678 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fee336b-4d8c-4696-a812-01e6dbbd629d" path="/var/lib/kubelet/pods/8fee336b-4d8c-4696-a812-01e6dbbd629d/volumes" Oct 30 21:43:36.567932 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:36.567869 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j4v8r_3df036ff-1c00-4ce2-9e86-8acbeedf6c82/ovn-acl-logging/0.log" Oct 30 21:43:36.569653 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:36.569628 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j4v8r_3df036ff-1c00-4ce2-9e86-8acbeedf6c82/ovn-acl-logging/0.log" Oct 30 21:43:39.956992 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:39.956956 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-91874e/distinct-2-w2-91874e-kserve-69fcf64f44-qm96r" Oct 30 21:43:39.976372 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:39.976341 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-91874e/distinct-2-w2-91874e-kserve-69fcf64f44-qm96r" Oct 30 21:43:40.004774 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:40.004744 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-91874e/distinct-2-w2-91874e-kserve-69fcf64f44-qm96r" Oct 30 21:43:41.533702 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:41.533662 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e9a6a20-7nl6f"] Oct 30 21:43:41.538835 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:41.538815 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9a6a20-7nl6f" Oct 30 21:43:41.596146 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:41.596116 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e9a6a20-7nl6f"] Oct 30 21:43:41.636548 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:41.636509 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wk8t\" (UniqueName: \"kubernetes.io/projected/badf2444-5704-4625-83df-a827672f494e-kube-api-access-6wk8t\") pod \"maas-api-revoke-sub-0e9a6a20-7nl6f\" (UID: \"badf2444-5704-4625-83df-a827672f494e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9a6a20-7nl6f" Oct 30 21:43:41.636742 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:41.636712 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/badf2444-5704-4625-83df-a827672f494e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e9a6a20-7nl6f\" (UID: \"badf2444-5704-4625-83df-a827672f494e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9a6a20-7nl6f" Oct 30 21:43:41.636837 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:41.636766 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/badf2444-5704-4625-83df-a827672f494e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e9a6a20-7nl6f\" (UID: \"badf2444-5704-4625-83df-a827672f494e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9a6a20-7nl6f" Oct 30 21:43:41.642519 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:41.642482 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:43:41.642700 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:41.642596 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:43:41.642700 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:41.642662 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:43:41.642700 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:41.642671 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 21:43:41.738378 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:41.738345 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wk8t\" (UniqueName: \"kubernetes.io/projected/badf2444-5704-4625-83df-a827672f494e-kube-api-access-6wk8t\") pod \"maas-api-revoke-sub-0e9a6a20-7nl6f\" (UID: \"badf2444-5704-4625-83df-a827672f494e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9a6a20-7nl6f" Oct 30 21:43:41.738574 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:41.738467 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/badf2444-5704-4625-83df-a827672f494e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e9a6a20-7nl6f\" (UID: \"badf2444-5704-4625-83df-a827672f494e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9a6a20-7nl6f" Oct 30 21:43:41.738574 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:41.738519 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/badf2444-5704-4625-83df-a827672f494e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e9a6a20-7nl6f\" (UID: \"badf2444-5704-4625-83df-a827672f494e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9a6a20-7nl6f" Oct 30 21:43:41.739201 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:41.739178 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/badf2444-5704-4625-83df-a827672f494e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e9a6a20-7nl6f\" (UID: \"badf2444-5704-4625-83df-a827672f494e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9a6a20-7nl6f" Oct 30 21:43:41.740962 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:41.740934 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/badf2444-5704-4625-83df-a827672f494e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e9a6a20-7nl6f\" (UID: \"badf2444-5704-4625-83df-a827672f494e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9a6a20-7nl6f" Oct 30 21:43:41.801861 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:41.801778 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wk8t\" (UniqueName: \"kubernetes.io/projected/badf2444-5704-4625-83df-a827672f494e-kube-api-access-6wk8t\") pod \"maas-api-revoke-sub-0e9a6a20-7nl6f\" (UID: \"badf2444-5704-4625-83df-a827672f494e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9a6a20-7nl6f" Oct 30 21:43:41.851791 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:41.851755 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9a6a20-7nl6f" Oct 30 21:43:42.229502 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:42.229469 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e9a6a20-7nl6f"] Oct 30 21:43:42.230523 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:43:42.230486 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbadf2444_5704_4625_83df_a827672f494e.slice/crio-18ebb7db1dea522f3e1c96b6a8fed475cc4c130c66251eba70ff17845b5d7af6 WatchSource:0}: Error finding container 18ebb7db1dea522f3e1c96b6a8fed475cc4c130c66251eba70ff17845b5d7af6: Status 404 returned error can't find the container with id 18ebb7db1dea522f3e1c96b6a8fed475cc4c130c66251eba70ff17845b5d7af6 Oct 30 21:43:43.015044 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:43.015003 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9a6a20-7nl6f" event={"ID":"badf2444-5704-4625-83df-a827672f494e","Type":"ContainerStarted","Data":"4326d9fc5b941ec3bba16215b9f0554c6687e4d9ac119659724315945a7b5632"} Oct 30 21:43:43.015456 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:43.015063 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9a6a20-7nl6f" event={"ID":"badf2444-5704-4625-83df-a827672f494e","Type":"ContainerStarted","Data":"18ebb7db1dea522f3e1c96b6a8fed475cc4c130c66251eba70ff17845b5d7af6"} Oct 30 21:43:43.134445 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:43.134384 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9a6a20-7nl6f" podStartSLOduration=2.134364385 podStartE2EDuration="2.134364385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:43.132289792 +0000 UTC m=+1206.874810728" watchObservedRunningTime="2026-10-30 21:43:43.134364385 +0000 UTC m=+1206.876885321" Oct 30 21:43:44.068545 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:44.067091 5748 generic.go:356] "Generic (PLEG): container finished" podID="badf2444-5704-4625-83df-a827672f494e" containerID="4326d9fc5b941ec3bba16215b9f0554c6687e4d9ac119659724315945a7b5632" exitCode=0 Oct 30 21:43:44.068545 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:44.067233 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9a6a20-7nl6f" event={"ID":"badf2444-5704-4625-83df-a827672f494e","Type":"ContainerDied","Data":"4326d9fc5b941ec3bba16215b9f0554c6687e4d9ac119659724315945a7b5632"} Oct 30 21:43:45.217351 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:45.217328 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9a6a20-7nl6f" Oct 30 21:43:45.284307 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:45.284255 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/badf2444-5704-4625-83df-a827672f494e-kube-api-access-6wk8t\" (UniqueName: \"kubernetes.io/projected/badf2444-5704-4625-83df-a827672f494e-kube-api-access-6wk8t\") pod \"badf2444-5704-4625-83df-a827672f494e\" (UID: \"badf2444-5704-4625-83df-a827672f494e\") " Oct 30 21:43:45.284472 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:45.284413 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/badf2444-5704-4625-83df-a827672f494e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/badf2444-5704-4625-83df-a827672f494e-maas-api-xks-ca\") pod \"badf2444-5704-4625-83df-a827672f494e\" (UID: \"badf2444-5704-4625-83df-a827672f494e\") " Oct 30 21:43:45.284554 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:45.284502 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/badf2444-5704-4625-83df-a827672f494e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/badf2444-5704-4625-83df-a827672f494e-maas-api-openshift-ca\") pod \"badf2444-5704-4625-83df-a827672f494e\" (UID: \"badf2444-5704-4625-83df-a827672f494e\") " Oct 30 21:43:45.284912 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:45.284871 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/badf2444-5704-4625-83df-a827672f494e-maas-api-openshift-ca" pod "badf2444-5704-4625-83df-a827672f494e" (UID: "badf2444-5704-4625-83df-a827672f494e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:43:45.286496 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:45.286468 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/badf2444-5704-4625-83df-a827672f494e-maas-api-xks-ca" pod "badf2444-5704-4625-83df-a827672f494e" (UID: "badf2444-5704-4625-83df-a827672f494e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:43:45.286940 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:45.286916 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/badf2444-5704-4625-83df-a827672f494e-kube-api-access-6wk8t" pod "badf2444-5704-4625-83df-a827672f494e" (UID: "badf2444-5704-4625-83df-a827672f494e"). InnerVolumeSpecName "kube-api-access-6wk8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:43:45.385639 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:45.385558 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/badf2444-5704-4625-83df-a827672f494e-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:43:45.385639 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:45.385589 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/badf2444-5704-4625-83df-a827672f494e-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:43:45.385639 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:45.385599 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wk8t\" (UniqueName: \"kubernetes.io/projected/badf2444-5704-4625-83df-a827672f494e-kube-api-access-6wk8t\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:43:46.083103 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:46.083075 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9a6a20-7nl6f" Oct 30 21:43:46.083103 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:46.083082 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9a6a20-7nl6f" event={"ID":"badf2444-5704-4625-83df-a827672f494e","Type":"ContainerDied","Data":"18ebb7db1dea522f3e1c96b6a8fed475cc4c130c66251eba70ff17845b5d7af6"} Oct 30 21:43:46.083314 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:46.083129 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18ebb7db1dea522f3e1c96b6a8fed475cc4c130c66251eba70ff17845b5d7af6" Oct 30 21:43:46.812871 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:46.812832 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0413d01a-65xkj"] Oct 30 21:43:46.813429 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:46.813412 5748 cpu_manager.go:414] "containerMap: removing container" podUID="badf2444-5704-4625-83df-a827672f494e" containerName="revoke-keys" Oct 30 21:43:46.813429 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:46.813428 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="badf2444-5704-4625-83df-a827672f494e" containerName="revoke-keys" Oct 30 21:43:46.813555 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:46.813544 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="badf2444-5704-4625-83df-a827672f494e" containerName="revoke-keys" Oct 30 21:43:46.817234 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:46.817215 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0413d01a-65xkj" Oct 30 21:43:46.873744 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:46.873709 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0413d01a-65xkj"] Oct 30 21:43:46.901250 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:46.901212 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t9mk\" (UniqueName: \"kubernetes.io/projected/62960998-49fb-43ff-aeb0-c1201517e2ab-kube-api-access-4t9mk\") pod \"maas-api-revoke-sub-0413d01a-65xkj\" (UID: \"62960998-49fb-43ff-aeb0-c1201517e2ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0413d01a-65xkj" Oct 30 21:43:46.901419 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:46.901353 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62960998-49fb-43ff-aeb0-c1201517e2ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0413d01a-65xkj\" (UID: \"62960998-49fb-43ff-aeb0-c1201517e2ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0413d01a-65xkj" Oct 30 21:43:46.901581 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:46.901562 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62960998-49fb-43ff-aeb0-c1201517e2ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0413d01a-65xkj\" (UID: \"62960998-49fb-43ff-aeb0-c1201517e2ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0413d01a-65xkj" Oct 30 21:43:46.919892 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:46.919865 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:43:46.920106 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:46.920007 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:43:46.920106 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:46.920050 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:43:46.920106 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:46.919865 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 21:43:47.002795 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:47.002759 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t9mk\" (UniqueName: \"kubernetes.io/projected/62960998-49fb-43ff-aeb0-c1201517e2ab-kube-api-access-4t9mk\") pod \"maas-api-revoke-sub-0413d01a-65xkj\" (UID: \"62960998-49fb-43ff-aeb0-c1201517e2ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0413d01a-65xkj" Oct 30 21:43:47.002990 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:47.002832 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62960998-49fb-43ff-aeb0-c1201517e2ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0413d01a-65xkj\" (UID: \"62960998-49fb-43ff-aeb0-c1201517e2ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0413d01a-65xkj" Oct 30 21:43:47.003091 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:47.003050 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62960998-49fb-43ff-aeb0-c1201517e2ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0413d01a-65xkj\" (UID: \"62960998-49fb-43ff-aeb0-c1201517e2ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0413d01a-65xkj" Oct 30 21:43:47.003543 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:47.003519 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62960998-49fb-43ff-aeb0-c1201517e2ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0413d01a-65xkj\" (UID: \"62960998-49fb-43ff-aeb0-c1201517e2ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0413d01a-65xkj" Oct 30 21:43:47.005471 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:47.005446 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62960998-49fb-43ff-aeb0-c1201517e2ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0413d01a-65xkj\" (UID: \"62960998-49fb-43ff-aeb0-c1201517e2ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0413d01a-65xkj" Oct 30 21:43:47.053631 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:47.053593 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc6702d8-x8np6"] Oct 30 21:43:47.059362 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:47.059335 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc6702d8-x8np6"] Oct 30 21:43:47.063416 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:47.063343 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9072d090-xgmg2"] Oct 30 21:43:47.065334 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:47.065308 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t9mk\" (UniqueName: \"kubernetes.io/projected/62960998-49fb-43ff-aeb0-c1201517e2ab-kube-api-access-4t9mk\") pod \"maas-api-revoke-sub-0413d01a-65xkj\" (UID: \"62960998-49fb-43ff-aeb0-c1201517e2ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0413d01a-65xkj" Oct 30 21:43:47.067020 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:47.066994 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9072d090-xgmg2"] Oct 30 21:43:47.130765 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:47.130732 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0413d01a-65xkj" Oct 30 21:43:47.510713 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:47.510683 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0413d01a-65xkj"] Oct 30 21:43:47.510986 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:43:47.510961 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62960998_49fb_43ff_aeb0_c1201517e2ab.slice/crio-8df575d4517ea5617e2d333dd5ca66bac9a4a09bab8486105829478b513b0d45 WatchSource:0}: Error finding container 8df575d4517ea5617e2d333dd5ca66bac9a4a09bab8486105829478b513b0d45: Status 404 returned error can't find the container with id 8df575d4517ea5617e2d333dd5ca66bac9a4a09bab8486105829478b513b0d45 Oct 30 21:43:48.049704 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:48.049665 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef54ebcb-29n9k"] Oct 30 21:43:48.054520 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:48.054476 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef54ebcb-29n9k"] Oct 30 21:43:48.098882 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:48.098795 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0413d01a-65xkj" event={"ID":"62960998-49fb-43ff-aeb0-c1201517e2ab","Type":"ContainerStarted","Data":"f772945a4c29d6192d596f916ede936039cb7f4901f0a9cddc90b7b3dd75675d"} Oct 30 21:43:48.098882 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:48.098854 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0413d01a-65xkj" event={"ID":"62960998-49fb-43ff-aeb0-c1201517e2ab","Type":"ContainerStarted","Data":"8df575d4517ea5617e2d333dd5ca66bac9a4a09bab8486105829478b513b0d45"} Oct 30 21:43:48.219115 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:48.219059 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0413d01a-65xkj" podStartSLOduration=2.219044818 podStartE2EDuration="2.219044818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:48.216109105 +0000 UTC m=+1211.958630041" watchObservedRunningTime="2026-10-30 21:43:48.219044818 +0000 UTC m=+1211.961565753" Oct 30 21:43:48.421022 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:48.420995 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38111ab1-6d6e-4392-8fd6-8fa44bbdf851" path="/var/lib/kubelet/pods/38111ab1-6d6e-4392-8fd6-8fa44bbdf851/volumes" Oct 30 21:43:48.421444 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:48.421430 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70a7ae14-0130-4733-ae4b-e40ce0597ac9" path="/var/lib/kubelet/pods/70a7ae14-0130-4733-ae4b-e40ce0597ac9/volumes" Oct 30 21:43:48.421805 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:48.421793 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cba56321-fa3c-4c0e-a7d7-7f89d9dcf866" path="/var/lib/kubelet/pods/cba56321-fa3c-4c0e-a7d7-7f89d9dcf866/volumes" Oct 30 21:43:49.107654 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:49.107621 5748 generic.go:356] "Generic (PLEG): container finished" podID="62960998-49fb-43ff-aeb0-c1201517e2ab" containerID="f772945a4c29d6192d596f916ede936039cb7f4901f0a9cddc90b7b3dd75675d" exitCode=0 Oct 30 21:43:49.108072 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:49.107698 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0413d01a-65xkj" event={"ID":"62960998-49fb-43ff-aeb0-c1201517e2ab","Type":"ContainerDied","Data":"f772945a4c29d6192d596f916ede936039cb7f4901f0a9cddc90b7b3dd75675d"} Oct 30 21:43:50.275670 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:50.275648 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0413d01a-65xkj" Oct 30 21:43:50.341709 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:50.341671 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62960998-49fb-43ff-aeb0-c1201517e2ab-kube-api-access-4t9mk\" (UniqueName: \"kubernetes.io/projected/62960998-49fb-43ff-aeb0-c1201517e2ab-kube-api-access-4t9mk\") pod \"62960998-49fb-43ff-aeb0-c1201517e2ab\" (UID: \"62960998-49fb-43ff-aeb0-c1201517e2ab\") " Oct 30 21:43:50.341886 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:50.341766 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62960998-49fb-43ff-aeb0-c1201517e2ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62960998-49fb-43ff-aeb0-c1201517e2ab-maas-api-openshift-ca\") pod \"62960998-49fb-43ff-aeb0-c1201517e2ab\" (UID: \"62960998-49fb-43ff-aeb0-c1201517e2ab\") " Oct 30 21:43:50.341886 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:50.341803 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62960998-49fb-43ff-aeb0-c1201517e2ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62960998-49fb-43ff-aeb0-c1201517e2ab-maas-api-xks-ca\") pod \"62960998-49fb-43ff-aeb0-c1201517e2ab\" (UID: \"62960998-49fb-43ff-aeb0-c1201517e2ab\") " Oct 30 21:43:50.342173 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:50.342148 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62960998-49fb-43ff-aeb0-c1201517e2ab-maas-api-openshift-ca" pod "62960998-49fb-43ff-aeb0-c1201517e2ab" (UID: "62960998-49fb-43ff-aeb0-c1201517e2ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:43:50.343734 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:50.343714 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62960998-49fb-43ff-aeb0-c1201517e2ab-kube-api-access-4t9mk" pod "62960998-49fb-43ff-aeb0-c1201517e2ab" (UID: "62960998-49fb-43ff-aeb0-c1201517e2ab"). InnerVolumeSpecName "kube-api-access-4t9mk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:43:50.343846 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:50.343826 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62960998-49fb-43ff-aeb0-c1201517e2ab-maas-api-xks-ca" pod "62960998-49fb-43ff-aeb0-c1201517e2ab" (UID: "62960998-49fb-43ff-aeb0-c1201517e2ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:43:50.443344 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:50.443308 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62960998-49fb-43ff-aeb0-c1201517e2ab-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:43:50.443344 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:50.443345 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62960998-49fb-43ff-aeb0-c1201517e2ab-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:43:50.443581 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:50.443371 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t9mk\" (UniqueName: \"kubernetes.io/projected/62960998-49fb-43ff-aeb0-c1201517e2ab-kube-api-access-4t9mk\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:43:51.123601 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:51.123569 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0413d01a-65xkj" Oct 30 21:43:51.123601 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:51.123581 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0413d01a-65xkj" event={"ID":"62960998-49fb-43ff-aeb0-c1201517e2ab","Type":"ContainerDied","Data":"8df575d4517ea5617e2d333dd5ca66bac9a4a09bab8486105829478b513b0d45"} Oct 30 21:43:51.123865 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:51.123628 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8df575d4517ea5617e2d333dd5ca66bac9a4a09bab8486105829478b513b0d45" Oct 30 21:43:51.420212 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:51.420175 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a51f0adb-x22tr"] Oct 30 21:43:51.420719 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:51.420705 5748 cpu_manager.go:414] "containerMap: removing container" podUID="62960998-49fb-43ff-aeb0-c1201517e2ab" containerName="revoke-keys" Oct 30 21:43:51.420763 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:51.420720 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62960998-49fb-43ff-aeb0-c1201517e2ab" containerName="revoke-keys" Oct 30 21:43:51.420857 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:51.420847 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="62960998-49fb-43ff-aeb0-c1201517e2ab" containerName="revoke-keys" Oct 30 21:43:51.424455 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:51.424439 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51f0adb-x22tr" Oct 30 21:43:51.454927 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:51.454870 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1a00bc7-6808-4176-b68c-c398ce86d586-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a51f0adb-x22tr\" (UID: \"f1a00bc7-6808-4176-b68c-c398ce86d586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51f0adb-x22tr" Oct 30 21:43:51.455111 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:51.455055 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxldg\" (UniqueName: \"kubernetes.io/projected/f1a00bc7-6808-4176-b68c-c398ce86d586-kube-api-access-xxldg\") pod \"maas-api-revoke-sub-a51f0adb-x22tr\" (UID: \"f1a00bc7-6808-4176-b68c-c398ce86d586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51f0adb-x22tr" Oct 30 21:43:51.455246 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:51.455225 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1a00bc7-6808-4176-b68c-c398ce86d586-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a51f0adb-x22tr\" (UID: \"f1a00bc7-6808-4176-b68c-c398ce86d586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51f0adb-x22tr" Oct 30 21:43:51.481278 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:51.481243 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:43:51.482548 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:51.482516 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a51f0adb-x22tr"] Oct 30 21:43:51.528623 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:51.528586 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:43:51.528623 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:51.528615 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 21:43:51.528829 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:51.528592 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:43:51.556760 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:51.556720 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1a00bc7-6808-4176-b68c-c398ce86d586-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a51f0adb-x22tr\" (UID: \"f1a00bc7-6808-4176-b68c-c398ce86d586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51f0adb-x22tr" Oct 30 21:43:51.556990 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:51.556815 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1a00bc7-6808-4176-b68c-c398ce86d586-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a51f0adb-x22tr\" (UID: \"f1a00bc7-6808-4176-b68c-c398ce86d586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51f0adb-x22tr" Oct 30 21:43:51.557050 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:51.557031 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxldg\" (UniqueName: \"kubernetes.io/projected/f1a00bc7-6808-4176-b68c-c398ce86d586-kube-api-access-xxldg\") pod \"maas-api-revoke-sub-a51f0adb-x22tr\" (UID: \"f1a00bc7-6808-4176-b68c-c398ce86d586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51f0adb-x22tr" Oct 30 21:43:51.557582 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:51.557556 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1a00bc7-6808-4176-b68c-c398ce86d586-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a51f0adb-x22tr\" (UID: \"f1a00bc7-6808-4176-b68c-c398ce86d586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51f0adb-x22tr" Oct 30 21:43:51.559320 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:51.559291 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1a00bc7-6808-4176-b68c-c398ce86d586-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a51f0adb-x22tr\" (UID: \"f1a00bc7-6808-4176-b68c-c398ce86d586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51f0adb-x22tr" Oct 30 21:43:51.618664 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:51.618628 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxldg\" (UniqueName: \"kubernetes.io/projected/f1a00bc7-6808-4176-b68c-c398ce86d586-kube-api-access-xxldg\") pod \"maas-api-revoke-sub-a51f0adb-x22tr\" (UID: \"f1a00bc7-6808-4176-b68c-c398ce86d586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51f0adb-x22tr" Oct 30 21:43:51.738095 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:51.738008 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51f0adb-x22tr" Oct 30 21:43:52.120131 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:52.120101 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a51f0adb-x22tr"] Oct 30 21:43:52.120957 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:43:52.120927 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1a00bc7_6808_4176_b68c_c398ce86d586.slice/crio-17bed1d5772629a1deee46c7df69256c7778f8087f803162db6b58a3b6819df7 WatchSource:0}: Error finding container 17bed1d5772629a1deee46c7df69256c7778f8087f803162db6b58a3b6819df7: Status 404 returned error can't find the container with id 17bed1d5772629a1deee46c7df69256c7778f8087f803162db6b58a3b6819df7 Oct 30 21:43:52.132495 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:52.132466 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51f0adb-x22tr" event={"ID":"f1a00bc7-6808-4176-b68c-c398ce86d586","Type":"ContainerStarted","Data":"17bed1d5772629a1deee46c7df69256c7778f8087f803162db6b58a3b6819df7"} Oct 30 21:43:53.151121 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:53.150840 5748 generic.go:356] "Generic (PLEG): container finished" podID="f1a00bc7-6808-4176-b68c-c398ce86d586" containerID="a506f884300f2f53a13446377bb2b173b17b41df9c559b7b4c28328cc58f11e2" exitCode=0 Oct 30 21:43:53.151583 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:53.151093 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51f0adb-x22tr" event={"ID":"f1a00bc7-6808-4176-b68c-c398ce86d586","Type":"ContainerDied","Data":"a506f884300f2f53a13446377bb2b173b17b41df9c559b7b4c28328cc58f11e2"} Oct 30 21:43:54.313056 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:54.313031 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51f0adb-x22tr" Oct 30 21:43:54.391292 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:54.391265 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1a00bc7-6808-4176-b68c-c398ce86d586-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1a00bc7-6808-4176-b68c-c398ce86d586-maas-api-openshift-ca\") pod \"f1a00bc7-6808-4176-b68c-c398ce86d586\" (UID: \"f1a00bc7-6808-4176-b68c-c398ce86d586\") " Oct 30 21:43:54.391401 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:54.391338 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1a00bc7-6808-4176-b68c-c398ce86d586-kube-api-access-xxldg\" (UniqueName: \"kubernetes.io/projected/f1a00bc7-6808-4176-b68c-c398ce86d586-kube-api-access-xxldg\") pod \"f1a00bc7-6808-4176-b68c-c398ce86d586\" (UID: \"f1a00bc7-6808-4176-b68c-c398ce86d586\") " Oct 30 21:43:54.391401 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:54.391365 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1a00bc7-6808-4176-b68c-c398ce86d586-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1a00bc7-6808-4176-b68c-c398ce86d586-maas-api-xks-ca\") pod \"f1a00bc7-6808-4176-b68c-c398ce86d586\" (UID: \"f1a00bc7-6808-4176-b68c-c398ce86d586\") " Oct 30 21:43:54.391691 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:54.391666 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1a00bc7-6808-4176-b68c-c398ce86d586-maas-api-openshift-ca" pod "f1a00bc7-6808-4176-b68c-c398ce86d586" (UID: "f1a00bc7-6808-4176-b68c-c398ce86d586"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:43:54.392098 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:54.392063 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1a00bc7-6808-4176-b68c-c398ce86d586-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:43:54.393752 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:54.393719 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1a00bc7-6808-4176-b68c-c398ce86d586-kube-api-access-xxldg" pod "f1a00bc7-6808-4176-b68c-c398ce86d586" (UID: "f1a00bc7-6808-4176-b68c-c398ce86d586"). InnerVolumeSpecName "kube-api-access-xxldg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:43:54.394013 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:54.393984 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1a00bc7-6808-4176-b68c-c398ce86d586-maas-api-xks-ca" pod "f1a00bc7-6808-4176-b68c-c398ce86d586" (UID: "f1a00bc7-6808-4176-b68c-c398ce86d586"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:43:54.493814 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:54.493774 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxldg\" (UniqueName: \"kubernetes.io/projected/f1a00bc7-6808-4176-b68c-c398ce86d586-kube-api-access-xxldg\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:43:54.493814 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:54.493815 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1a00bc7-6808-4176-b68c-c398ce86d586-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:43:55.167031 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:55.166990 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51f0adb-x22tr" event={"ID":"f1a00bc7-6808-4176-b68c-c398ce86d586","Type":"ContainerDied","Data":"17bed1d5772629a1deee46c7df69256c7778f8087f803162db6b58a3b6819df7"} Oct 30 21:43:55.167226 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:55.167063 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17bed1d5772629a1deee46c7df69256c7778f8087f803162db6b58a3b6819df7" Oct 30 21:43:55.167226 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:43:55.167004 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51f0adb-x22tr" Oct 30 21:44:00.150040 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:44:00.149997 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:44:02.458841 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:02.458808 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf2dd5b6-k54j2"] Oct 30 21:44:02.459370 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:02.459356 5748 cpu_manager.go:414] "containerMap: removing container" podUID="f1a00bc7-6808-4176-b68c-c398ce86d586" containerName="revoke-keys" Oct 30 21:44:02.459414 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:02.459371 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1a00bc7-6808-4176-b68c-c398ce86d586" containerName="revoke-keys" Oct 30 21:44:02.459472 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:02.459463 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1a00bc7-6808-4176-b68c-c398ce86d586" containerName="revoke-keys" Oct 30 21:44:02.463229 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:02.463210 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf2dd5b6-k54j2" Oct 30 21:44:02.522206 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:02.522178 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf2dd5b6-k54j2"] Oct 30 21:44:02.567881 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:02.567845 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 21:44:02.567881 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:02.567869 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:44:02.568132 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:02.567845 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:44:02.568132 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:02.567923 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:44:02.593520 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:02.593481 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxkkj\" (UniqueName: \"kubernetes.io/projected/c568b44c-1112-4bea-befa-a58118e54637-kube-api-access-zxkkj\") pod \"maas-api-revoke-sub-cf2dd5b6-k54j2\" (UID: \"c568b44c-1112-4bea-befa-a58118e54637\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf2dd5b6-k54j2" Oct 30 21:44:02.593680 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:02.593609 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c568b44c-1112-4bea-befa-a58118e54637-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf2dd5b6-k54j2\" (UID: \"c568b44c-1112-4bea-befa-a58118e54637\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf2dd5b6-k54j2" Oct 30 21:44:02.593721 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:02.593688 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c568b44c-1112-4bea-befa-a58118e54637-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf2dd5b6-k54j2\" (UID: \"c568b44c-1112-4bea-befa-a58118e54637\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf2dd5b6-k54j2" Oct 30 21:44:02.695259 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:02.695222 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxkkj\" (UniqueName: \"kubernetes.io/projected/c568b44c-1112-4bea-befa-a58118e54637-kube-api-access-zxkkj\") pod \"maas-api-revoke-sub-cf2dd5b6-k54j2\" (UID: \"c568b44c-1112-4bea-befa-a58118e54637\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf2dd5b6-k54j2" Oct 30 21:44:02.695475 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:02.695307 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c568b44c-1112-4bea-befa-a58118e54637-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf2dd5b6-k54j2\" (UID: \"c568b44c-1112-4bea-befa-a58118e54637\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf2dd5b6-k54j2" Oct 30 21:44:02.695475 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:02.695362 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c568b44c-1112-4bea-befa-a58118e54637-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf2dd5b6-k54j2\" (UID: \"c568b44c-1112-4bea-befa-a58118e54637\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf2dd5b6-k54j2" Oct 30 21:44:02.696075 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:02.696047 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c568b44c-1112-4bea-befa-a58118e54637-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf2dd5b6-k54j2\" (UID: \"c568b44c-1112-4bea-befa-a58118e54637\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf2dd5b6-k54j2" Oct 30 21:44:02.697730 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:02.697703 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c568b44c-1112-4bea-befa-a58118e54637-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf2dd5b6-k54j2\" (UID: \"c568b44c-1112-4bea-befa-a58118e54637\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf2dd5b6-k54j2" Oct 30 21:44:02.757770 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:02.757694 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxkkj\" (UniqueName: \"kubernetes.io/projected/c568b44c-1112-4bea-befa-a58118e54637-kube-api-access-zxkkj\") pod \"maas-api-revoke-sub-cf2dd5b6-k54j2\" (UID: \"c568b44c-1112-4bea-befa-a58118e54637\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf2dd5b6-k54j2" Oct 30 21:44:02.776723 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:02.776694 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf2dd5b6-k54j2" Oct 30 21:44:03.050612 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:03.050533 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d13b782e-2zxh4"] Oct 30 21:44:03.054016 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:03.053992 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d13b782e-2zxh4"] Oct 30 21:44:03.169233 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:03.169203 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf2dd5b6-k54j2"] Oct 30 21:44:03.170389 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:44:03.170352 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc568b44c_1112_4bea_befa_a58118e54637.slice/crio-b36f94acaa04259ec4b744d07317cc6e446792e99bb2fcdbb3e1a6d5a38c7c3a WatchSource:0}: Error finding container b36f94acaa04259ec4b744d07317cc6e446792e99bb2fcdbb3e1a6d5a38c7c3a: Status 404 returned error can't find the container with id b36f94acaa04259ec4b744d07317cc6e446792e99bb2fcdbb3e1a6d5a38c7c3a Oct 30 21:44:03.224779 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:03.224749 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf2dd5b6-k54j2" event={"ID":"c568b44c-1112-4bea-befa-a58118e54637","Type":"ContainerStarted","Data":"b36f94acaa04259ec4b744d07317cc6e446792e99bb2fcdbb3e1a6d5a38c7c3a"} Oct 30 21:44:04.049278 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:04.049242 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62098d29-cggjg"] Oct 30 21:44:04.099771 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:04.099736 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62098d29-cggjg"] Oct 30 21:44:04.233358 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:04.233320 5748 generic.go:356] "Generic (PLEG): container finished" podID="c568b44c-1112-4bea-befa-a58118e54637" containerID="ff096887438d8fac25669cb0039ac32ec8bf0bb9d08c6f853bb48a322740808f" exitCode=0 Oct 30 21:44:04.233537 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:04.233401 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf2dd5b6-k54j2" event={"ID":"c568b44c-1112-4bea-befa-a58118e54637","Type":"ContainerDied","Data":"ff096887438d8fac25669cb0039ac32ec8bf0bb9d08c6f853bb48a322740808f"} Oct 30 21:44:04.417365 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:04.417333 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82a501af-15ab-4fd6-b6bf-38e89b5c1727" path="/var/lib/kubelet/pods/82a501af-15ab-4fd6-b6bf-38e89b5c1727/volumes" Oct 30 21:44:04.417747 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:04.417734 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c463f776-084f-43de-98f1-1035dcc8d43a" path="/var/lib/kubelet/pods/c463f776-084f-43de-98f1-1035dcc8d43a/volumes" Oct 30 21:44:05.395714 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:05.395689 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf2dd5b6-k54j2" Oct 30 21:44:05.530228 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:05.530148 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c568b44c-1112-4bea-befa-a58118e54637-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c568b44c-1112-4bea-befa-a58118e54637-maas-api-xks-ca\") pod \"c568b44c-1112-4bea-befa-a58118e54637\" (UID: \"c568b44c-1112-4bea-befa-a58118e54637\") " Oct 30 21:44:05.530405 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:05.530247 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c568b44c-1112-4bea-befa-a58118e54637-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c568b44c-1112-4bea-befa-a58118e54637-maas-api-openshift-ca\") pod \"c568b44c-1112-4bea-befa-a58118e54637\" (UID: \"c568b44c-1112-4bea-befa-a58118e54637\") " Oct 30 21:44:05.530405 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:05.530284 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c568b44c-1112-4bea-befa-a58118e54637-kube-api-access-zxkkj\" (UniqueName: \"kubernetes.io/projected/c568b44c-1112-4bea-befa-a58118e54637-kube-api-access-zxkkj\") pod \"c568b44c-1112-4bea-befa-a58118e54637\" (UID: \"c568b44c-1112-4bea-befa-a58118e54637\") " Oct 30 21:44:05.530655 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:05.530625 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c568b44c-1112-4bea-befa-a58118e54637-maas-api-openshift-ca" pod "c568b44c-1112-4bea-befa-a58118e54637" (UID: "c568b44c-1112-4bea-befa-a58118e54637"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:44:05.531241 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:05.531185 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c568b44c-1112-4bea-befa-a58118e54637-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:05.532586 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:05.532557 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c568b44c-1112-4bea-befa-a58118e54637-kube-api-access-zxkkj" pod "c568b44c-1112-4bea-befa-a58118e54637" (UID: "c568b44c-1112-4bea-befa-a58118e54637"). InnerVolumeSpecName "kube-api-access-zxkkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:05.532658 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:05.532628 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c568b44c-1112-4bea-befa-a58118e54637-maas-api-xks-ca" pod "c568b44c-1112-4bea-befa-a58118e54637" (UID: "c568b44c-1112-4bea-befa-a58118e54637"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:44:05.631960 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:05.631917 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c568b44c-1112-4bea-befa-a58118e54637-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:05.631960 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:05.631955 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxkkj\" (UniqueName: \"kubernetes.io/projected/c568b44c-1112-4bea-befa-a58118e54637-kube-api-access-zxkkj\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:06.249315 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:06.249281 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf2dd5b6-k54j2" Oct 30 21:44:06.249512 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:06.249321 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf2dd5b6-k54j2" event={"ID":"c568b44c-1112-4bea-befa-a58118e54637","Type":"ContainerDied","Data":"b36f94acaa04259ec4b744d07317cc6e446792e99bb2fcdbb3e1a6d5a38c7c3a"} Oct 30 21:44:06.249512 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:06.249408 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b36f94acaa04259ec4b744d07317cc6e446792e99bb2fcdbb3e1a6d5a38c7c3a" Oct 30 21:44:07.077374 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:07.077337 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd7fe367-chq4v"] Oct 30 21:44:07.077925 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:07.077887 5748 cpu_manager.go:414] "containerMap: removing container" podUID="c568b44c-1112-4bea-befa-a58118e54637" containerName="revoke-keys" Oct 30 21:44:07.077925 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:07.077920 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c568b44c-1112-4bea-befa-a58118e54637" containerName="revoke-keys" Oct 30 21:44:07.078054 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:07.078045 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="c568b44c-1112-4bea-befa-a58118e54637" containerName="revoke-keys" Oct 30 21:44:07.081494 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:07.081477 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd7fe367-chq4v" Oct 30 21:44:07.139947 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:07.139913 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd7fe367-chq4v"] Oct 30 21:44:07.186334 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:07.186305 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:44:07.186334 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:07.186326 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:44:07.186541 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:07.186331 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:44:07.186541 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:07.186441 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 21:44:07.251925 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:07.251862 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p5k7\" (UniqueName: \"kubernetes.io/projected/d121da3d-7c63-4cf8-8fc6-7c87818ccc8c-kube-api-access-4p5k7\") pod \"maas-api-revoke-sub-bd7fe367-chq4v\" (UID: \"d121da3d-7c63-4cf8-8fc6-7c87818ccc8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd7fe367-chq4v" Oct 30 21:44:07.252132 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:07.252076 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d121da3d-7c63-4cf8-8fc6-7c87818ccc8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd7fe367-chq4v\" (UID: \"d121da3d-7c63-4cf8-8fc6-7c87818ccc8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd7fe367-chq4v" Oct 30 21:44:07.252205 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:07.252172 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d121da3d-7c63-4cf8-8fc6-7c87818ccc8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd7fe367-chq4v\" (UID: \"d121da3d-7c63-4cf8-8fc6-7c87818ccc8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd7fe367-chq4v" Oct 30 21:44:07.353573 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:07.353485 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d121da3d-7c63-4cf8-8fc6-7c87818ccc8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd7fe367-chq4v\" (UID: \"d121da3d-7c63-4cf8-8fc6-7c87818ccc8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd7fe367-chq4v" Oct 30 21:44:07.353753 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:07.353589 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p5k7\" (UniqueName: \"kubernetes.io/projected/d121da3d-7c63-4cf8-8fc6-7c87818ccc8c-kube-api-access-4p5k7\") pod \"maas-api-revoke-sub-bd7fe367-chq4v\" (UID: \"d121da3d-7c63-4cf8-8fc6-7c87818ccc8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd7fe367-chq4v" Oct 30 21:44:07.353753 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:07.353716 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d121da3d-7c63-4cf8-8fc6-7c87818ccc8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd7fe367-chq4v\" (UID: \"d121da3d-7c63-4cf8-8fc6-7c87818ccc8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd7fe367-chq4v" Oct 30 21:44:07.354309 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:07.354282 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d121da3d-7c63-4cf8-8fc6-7c87818ccc8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd7fe367-chq4v\" (UID: \"d121da3d-7c63-4cf8-8fc6-7c87818ccc8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd7fe367-chq4v" Oct 30 21:44:07.355931 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:07.355880 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d121da3d-7c63-4cf8-8fc6-7c87818ccc8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd7fe367-chq4v\" (UID: \"d121da3d-7c63-4cf8-8fc6-7c87818ccc8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd7fe367-chq4v" Oct 30 21:44:07.416191 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:07.416161 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p5k7\" (UniqueName: \"kubernetes.io/projected/d121da3d-7c63-4cf8-8fc6-7c87818ccc8c-kube-api-access-4p5k7\") pod \"maas-api-revoke-sub-bd7fe367-chq4v\" (UID: \"d121da3d-7c63-4cf8-8fc6-7c87818ccc8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd7fe367-chq4v" Oct 30 21:44:07.694823 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:07.694789 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd7fe367-chq4v" Oct 30 21:44:08.049926 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:08.049806 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-733e3b40-v6dsk"] Oct 30 21:44:08.100420 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:08.100385 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-733e3b40-v6dsk"] Oct 30 21:44:08.103438 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:08.103409 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd7fe367-chq4v"] Oct 30 21:44:08.112253 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:44:08.112221 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd121da3d_7c63_4cf8_8fc6_7c87818ccc8c.slice/crio-7cce616e890a986954b2f44907c35debbe6b60b48863fc8fa17c2779a900cbf3 WatchSource:0}: Error finding container 7cce616e890a986954b2f44907c35debbe6b60b48863fc8fa17c2779a900cbf3: Status 404 returned error can't find the container with id 7cce616e890a986954b2f44907c35debbe6b60b48863fc8fa17c2779a900cbf3 Oct 30 21:44:08.274926 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:08.274859 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd7fe367-chq4v" event={"ID":"d121da3d-7c63-4cf8-8fc6-7c87818ccc8c","Type":"ContainerStarted","Data":"7cce616e890a986954b2f44907c35debbe6b60b48863fc8fa17c2779a900cbf3"} Oct 30 21:44:08.417807 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:08.417777 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a47cdb8d-3422-4853-b26e-6aa87fedfb4b" path="/var/lib/kubelet/pods/a47cdb8d-3422-4853-b26e-6aa87fedfb4b/volumes" Oct 30 21:44:09.283674 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:09.283582 5748 generic.go:356] "Generic (PLEG): container finished" podID="d121da3d-7c63-4cf8-8fc6-7c87818ccc8c" containerID="e80272bdf0b87eb5eccdb62007e6ee67761ed211bfeaddc7ae5f429a7a2cb6f6" exitCode=0 Oct 30 21:44:09.284079 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:09.283676 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd7fe367-chq4v" event={"ID":"d121da3d-7c63-4cf8-8fc6-7c87818ccc8c","Type":"ContainerDied","Data":"e80272bdf0b87eb5eccdb62007e6ee67761ed211bfeaddc7ae5f429a7a2cb6f6"} Oct 30 21:44:10.451051 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:10.451026 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd7fe367-chq4v" Oct 30 21:44:10.493111 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:10.493075 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d121da3d-7c63-4cf8-8fc6-7c87818ccc8c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d121da3d-7c63-4cf8-8fc6-7c87818ccc8c-maas-api-openshift-ca\") pod \"d121da3d-7c63-4cf8-8fc6-7c87818ccc8c\" (UID: \"d121da3d-7c63-4cf8-8fc6-7c87818ccc8c\") " Oct 30 21:44:10.493257 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:10.493146 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d121da3d-7c63-4cf8-8fc6-7c87818ccc8c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d121da3d-7c63-4cf8-8fc6-7c87818ccc8c-maas-api-xks-ca\") pod \"d121da3d-7c63-4cf8-8fc6-7c87818ccc8c\" (UID: \"d121da3d-7c63-4cf8-8fc6-7c87818ccc8c\") " Oct 30 21:44:10.493331 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:10.493308 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d121da3d-7c63-4cf8-8fc6-7c87818ccc8c-kube-api-access-4p5k7\" (UniqueName: \"kubernetes.io/projected/d121da3d-7c63-4cf8-8fc6-7c87818ccc8c-kube-api-access-4p5k7\") pod \"d121da3d-7c63-4cf8-8fc6-7c87818ccc8c\" (UID: \"d121da3d-7c63-4cf8-8fc6-7c87818ccc8c\") " Oct 30 21:44:10.493567 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:10.493529 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d121da3d-7c63-4cf8-8fc6-7c87818ccc8c-maas-api-openshift-ca" pod "d121da3d-7c63-4cf8-8fc6-7c87818ccc8c" (UID: "d121da3d-7c63-4cf8-8fc6-7c87818ccc8c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:44:10.494140 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:10.493922 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d121da3d-7c63-4cf8-8fc6-7c87818ccc8c-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:10.495560 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:10.495530 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d121da3d-7c63-4cf8-8fc6-7c87818ccc8c-kube-api-access-4p5k7" pod "d121da3d-7c63-4cf8-8fc6-7c87818ccc8c" (UID: "d121da3d-7c63-4cf8-8fc6-7c87818ccc8c"). InnerVolumeSpecName "kube-api-access-4p5k7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:10.495696 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:10.495666 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d121da3d-7c63-4cf8-8fc6-7c87818ccc8c-maas-api-xks-ca" pod "d121da3d-7c63-4cf8-8fc6-7c87818ccc8c" (UID: "d121da3d-7c63-4cf8-8fc6-7c87818ccc8c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:44:10.595235 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:10.595152 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d121da3d-7c63-4cf8-8fc6-7c87818ccc8c-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:10.595235 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:10.595180 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4p5k7\" (UniqueName: \"kubernetes.io/projected/d121da3d-7c63-4cf8-8fc6-7c87818ccc8c-kube-api-access-4p5k7\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:11.050950 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:11.050915 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d96d4216-9zkrg"] Oct 30 21:44:11.054282 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:11.054255 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d96d4216-9zkrg"] Oct 30 21:44:11.103147 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:11.103111 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09ae27cf-jqpxr"] Oct 30 21:44:11.107732 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:11.107699 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09ae27cf-jqpxr"] Oct 30 21:44:11.298417 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:11.298381 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd7fe367-chq4v" event={"ID":"d121da3d-7c63-4cf8-8fc6-7c87818ccc8c","Type":"ContainerDied","Data":"7cce616e890a986954b2f44907c35debbe6b60b48863fc8fa17c2779a900cbf3"} Oct 30 21:44:11.298627 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:11.298429 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7cce616e890a986954b2f44907c35debbe6b60b48863fc8fa17c2779a900cbf3" Oct 30 21:44:11.298627 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:11.298396 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd7fe367-chq4v" Oct 30 21:44:11.323679 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:11.323651 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0ea77a6-pf7qk"] Oct 30 21:44:11.324230 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:11.324216 5748 cpu_manager.go:414] "containerMap: removing container" podUID="d121da3d-7c63-4cf8-8fc6-7c87818ccc8c" containerName="revoke-keys" Oct 30 21:44:11.324285 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:11.324231 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d121da3d-7c63-4cf8-8fc6-7c87818ccc8c" containerName="revoke-keys" Oct 30 21:44:11.324364 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:11.324354 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="d121da3d-7c63-4cf8-8fc6-7c87818ccc8c" containerName="revoke-keys" Oct 30 21:44:11.327873 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:11.327856 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0ea77a6-pf7qk" Oct 30 21:44:11.385139 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:11.385092 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:44:11.385427 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:11.385081 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:44:11.385427 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:11.385210 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:44:11.385427 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:11.385282 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 21:44:11.388044 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:11.388020 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0ea77a6-pf7qk"] Oct 30 21:44:11.405534 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:11.405504 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0749f3f-6b1e-4e78-9ab2-9d4c8b65e46e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0ea77a6-pf7qk\" (UID: \"b0749f3f-6b1e-4e78-9ab2-9d4c8b65e46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0ea77a6-pf7qk" Oct 30 21:44:11.405652 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:11.405609 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfsb7\" (UniqueName: \"kubernetes.io/projected/b0749f3f-6b1e-4e78-9ab2-9d4c8b65e46e-kube-api-access-wfsb7\") pod \"maas-api-revoke-sub-b0ea77a6-pf7qk\" (UID: \"b0749f3f-6b1e-4e78-9ab2-9d4c8b65e46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0ea77a6-pf7qk" Oct 30 21:44:11.405700 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:11.405678 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0749f3f-6b1e-4e78-9ab2-9d4c8b65e46e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0ea77a6-pf7qk\" (UID: \"b0749f3f-6b1e-4e78-9ab2-9d4c8b65e46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0ea77a6-pf7qk" Oct 30 21:44:11.507211 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:11.507172 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0749f3f-6b1e-4e78-9ab2-9d4c8b65e46e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0ea77a6-pf7qk\" (UID: \"b0749f3f-6b1e-4e78-9ab2-9d4c8b65e46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0ea77a6-pf7qk" Oct 30 21:44:11.507620 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:11.507292 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0749f3f-6b1e-4e78-9ab2-9d4c8b65e46e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0ea77a6-pf7qk\" (UID: \"b0749f3f-6b1e-4e78-9ab2-9d4c8b65e46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0ea77a6-pf7qk" Oct 30 21:44:11.507620 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:11.507381 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfsb7\" (UniqueName: \"kubernetes.io/projected/b0749f3f-6b1e-4e78-9ab2-9d4c8b65e46e-kube-api-access-wfsb7\") pod \"maas-api-revoke-sub-b0ea77a6-pf7qk\" (UID: \"b0749f3f-6b1e-4e78-9ab2-9d4c8b65e46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0ea77a6-pf7qk" Oct 30 21:44:11.507915 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:11.507876 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0749f3f-6b1e-4e78-9ab2-9d4c8b65e46e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0ea77a6-pf7qk\" (UID: \"b0749f3f-6b1e-4e78-9ab2-9d4c8b65e46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0ea77a6-pf7qk" Oct 30 21:44:11.509748 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:11.509712 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0749f3f-6b1e-4e78-9ab2-9d4c8b65e46e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0ea77a6-pf7qk\" (UID: \"b0749f3f-6b1e-4e78-9ab2-9d4c8b65e46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0ea77a6-pf7qk" Oct 30 21:44:11.569560 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:11.569487 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfsb7\" (UniqueName: \"kubernetes.io/projected/b0749f3f-6b1e-4e78-9ab2-9d4c8b65e46e-kube-api-access-wfsb7\") pod \"maas-api-revoke-sub-b0ea77a6-pf7qk\" (UID: \"b0749f3f-6b1e-4e78-9ab2-9d4c8b65e46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0ea77a6-pf7qk" Oct 30 21:44:11.640503 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:11.640476 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0ea77a6-pf7qk" Oct 30 21:44:12.021138 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:12.021106 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0ea77a6-pf7qk"] Oct 30 21:44:12.022033 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:44:12.022005 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0749f3f_6b1e_4e78_9ab2_9d4c8b65e46e.slice/crio-8da8758afc1a83ecd23c4d0013f51cdd3249547818a344c3818f24c2bc10fb6c WatchSource:0}: Error finding container 8da8758afc1a83ecd23c4d0013f51cdd3249547818a344c3818f24c2bc10fb6c: Status 404 returned error can't find the container with id 8da8758afc1a83ecd23c4d0013f51cdd3249547818a344c3818f24c2bc10fb6c Oct 30 21:44:12.307406 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:12.307367 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0ea77a6-pf7qk" event={"ID":"b0749f3f-6b1e-4e78-9ab2-9d4c8b65e46e","Type":"ContainerStarted","Data":"8da8758afc1a83ecd23c4d0013f51cdd3249547818a344c3818f24c2bc10fb6c"} Oct 30 21:44:12.417784 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:12.417750 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1597353f-0050-4a88-89df-ccec34a0e2b0" path="/var/lib/kubelet/pods/1597353f-0050-4a88-89df-ccec34a0e2b0/volumes" Oct 30 21:44:12.418253 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:12.418236 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f2cd81a-d648-4baf-94c7-24ffeadce3d7" path="/var/lib/kubelet/pods/2f2cd81a-d648-4baf-94c7-24ffeadce3d7/volumes" Oct 30 21:44:13.317049 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:13.316952 5748 generic.go:356] "Generic (PLEG): container finished" podID="b0749f3f-6b1e-4e78-9ab2-9d4c8b65e46e" containerID="9ef24eb689b54cd6c4eb51440ea1594c60594f760cb6ea507385a25b6c372a36" exitCode=0 Oct 30 21:44:13.317489 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:13.317033 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0ea77a6-pf7qk" event={"ID":"b0749f3f-6b1e-4e78-9ab2-9d4c8b65e46e","Type":"ContainerDied","Data":"9ef24eb689b54cd6c4eb51440ea1594c60594f760cb6ea507385a25b6c372a36"} Oct 30 21:44:14.474598 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:14.474568 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0ea77a6-pf7qk" Oct 30 21:44:14.547110 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:14.547074 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0749f3f-6b1e-4e78-9ab2-9d4c8b65e46e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0749f3f-6b1e-4e78-9ab2-9d4c8b65e46e-maas-api-openshift-ca\") pod \"b0749f3f-6b1e-4e78-9ab2-9d4c8b65e46e\" (UID: \"b0749f3f-6b1e-4e78-9ab2-9d4c8b65e46e\") " Oct 30 21:44:14.547289 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:14.547188 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0749f3f-6b1e-4e78-9ab2-9d4c8b65e46e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0749f3f-6b1e-4e78-9ab2-9d4c8b65e46e-maas-api-xks-ca\") pod \"b0749f3f-6b1e-4e78-9ab2-9d4c8b65e46e\" (UID: \"b0749f3f-6b1e-4e78-9ab2-9d4c8b65e46e\") " Oct 30 21:44:14.547364 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:14.547347 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0749f3f-6b1e-4e78-9ab2-9d4c8b65e46e-kube-api-access-wfsb7\" (UniqueName: \"kubernetes.io/projected/b0749f3f-6b1e-4e78-9ab2-9d4c8b65e46e-kube-api-access-wfsb7\") pod \"b0749f3f-6b1e-4e78-9ab2-9d4c8b65e46e\" (UID: \"b0749f3f-6b1e-4e78-9ab2-9d4c8b65e46e\") " Oct 30 21:44:14.547437 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:14.547413 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0749f3f-6b1e-4e78-9ab2-9d4c8b65e46e-maas-api-openshift-ca" pod "b0749f3f-6b1e-4e78-9ab2-9d4c8b65e46e" (UID: "b0749f3f-6b1e-4e78-9ab2-9d4c8b65e46e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:44:14.547798 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:14.547779 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0749f3f-6b1e-4e78-9ab2-9d4c8b65e46e-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:14.549257 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:14.549237 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0749f3f-6b1e-4e78-9ab2-9d4c8b65e46e-maas-api-xks-ca" pod "b0749f3f-6b1e-4e78-9ab2-9d4c8b65e46e" (UID: "b0749f3f-6b1e-4e78-9ab2-9d4c8b65e46e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:44:14.549429 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:14.549408 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0749f3f-6b1e-4e78-9ab2-9d4c8b65e46e-kube-api-access-wfsb7" pod "b0749f3f-6b1e-4e78-9ab2-9d4c8b65e46e" (UID: "b0749f3f-6b1e-4e78-9ab2-9d4c8b65e46e"). InnerVolumeSpecName "kube-api-access-wfsb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:14.649148 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:14.649115 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0749f3f-6b1e-4e78-9ab2-9d4c8b65e46e-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:14.649148 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:14.649145 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfsb7\" (UniqueName: \"kubernetes.io/projected/b0749f3f-6b1e-4e78-9ab2-9d4c8b65e46e-kube-api-access-wfsb7\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:15.332794 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:15.332764 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0ea77a6-pf7qk" Oct 30 21:44:15.333031 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:15.332797 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0ea77a6-pf7qk" event={"ID":"b0749f3f-6b1e-4e78-9ab2-9d4c8b65e46e","Type":"ContainerDied","Data":"8da8758afc1a83ecd23c4d0013f51cdd3249547818a344c3818f24c2bc10fb6c"} Oct 30 21:44:15.333031 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:15.332851 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8da8758afc1a83ecd23c4d0013f51cdd3249547818a344c3818f24c2bc10fb6c" Oct 30 21:44:16.549131 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:16.549091 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh"] Oct 30 21:44:16.549976 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:16.549951 5748 cpu_manager.go:414] "containerMap: removing container" podUID="b0749f3f-6b1e-4e78-9ab2-9d4c8b65e46e" containerName="revoke-keys" Oct 30 21:44:16.550090 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:16.549978 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0749f3f-6b1e-4e78-9ab2-9d4c8b65e46e" containerName="revoke-keys" Oct 30 21:44:16.550343 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:16.550173 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0749f3f-6b1e-4e78-9ab2-9d4c8b65e46e" containerName="revoke-keys" Oct 30 21:44:16.556038 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:16.556012 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh" Oct 30 21:44:16.646997 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:16.646959 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh"] Oct 30 21:44:16.674662 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:16.674630 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6c9d2bc-60d6-494c-af57-574e85c4861c-workload-socket\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh\" (UID: \"b6c9d2bc-60d6-494c-af57-574e85c4861c\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh" Oct 30 21:44:16.674662 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:16.674664 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6c9d2bc-60d6-494c-af57-574e85c4861c-istio-podinfo\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh\" (UID: \"b6c9d2bc-60d6-494c-af57-574e85c4861c\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh" Oct 30 21:44:16.674945 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:16.674703 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6c9d2bc-60d6-494c-af57-574e85c4861c-credential-socket\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh\" (UID: \"b6c9d2bc-60d6-494c-af57-574e85c4861c\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh" Oct 30 21:44:16.674945 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:16.674879 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6c9d2bc-60d6-494c-af57-574e85c4861c-istio-token\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh\" (UID: \"b6c9d2bc-60d6-494c-af57-574e85c4861c\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh" Oct 30 21:44:16.675069 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:16.675042 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqjhf\" (UniqueName: \"kubernetes.io/projected/b6c9d2bc-60d6-494c-af57-574e85c4861c-kube-api-access-tqjhf\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh\" (UID: \"b6c9d2bc-60d6-494c-af57-574e85c4861c\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh" Oct 30 21:44:16.675132 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:16.675094 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6c9d2bc-60d6-494c-af57-574e85c4861c-workload-certs\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh\" (UID: \"b6c9d2bc-60d6-494c-af57-574e85c4861c\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh" Oct 30 21:44:16.675189 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:16.675126 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6c9d2bc-60d6-494c-af57-574e85c4861c-istiod-ca-cert\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh\" (UID: \"b6c9d2bc-60d6-494c-af57-574e85c4861c\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh" Oct 30 21:44:16.675189 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:16.675160 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6c9d2bc-60d6-494c-af57-574e85c4861c-istio-data\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh\" (UID: \"b6c9d2bc-60d6-494c-af57-574e85c4861c\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh" Oct 30 21:44:16.675307 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:16.675203 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6c9d2bc-60d6-494c-af57-574e85c4861c-istio-envoy\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh\" (UID: \"b6c9d2bc-60d6-494c-af57-574e85c4861c\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh" Oct 30 21:44:16.776679 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:16.776640 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6c9d2bc-60d6-494c-af57-574e85c4861c-istio-token\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh\" (UID: \"b6c9d2bc-60d6-494c-af57-574e85c4861c\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh" Oct 30 21:44:16.776866 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:16.776739 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqjhf\" (UniqueName: \"kubernetes.io/projected/b6c9d2bc-60d6-494c-af57-574e85c4861c-kube-api-access-tqjhf\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh\" (UID: \"b6c9d2bc-60d6-494c-af57-574e85c4861c\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh" Oct 30 21:44:16.776866 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:16.776765 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6c9d2bc-60d6-494c-af57-574e85c4861c-workload-certs\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh\" (UID: \"b6c9d2bc-60d6-494c-af57-574e85c4861c\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh" Oct 30 21:44:16.776866 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:16.776782 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6c9d2bc-60d6-494c-af57-574e85c4861c-istiod-ca-cert\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh\" (UID: \"b6c9d2bc-60d6-494c-af57-574e85c4861c\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh" Oct 30 21:44:16.776866 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:16.776807 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6c9d2bc-60d6-494c-af57-574e85c4861c-istio-data\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh\" (UID: \"b6c9d2bc-60d6-494c-af57-574e85c4861c\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh" Oct 30 21:44:16.776866 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:16.776841 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6c9d2bc-60d6-494c-af57-574e85c4861c-istio-envoy\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh\" (UID: \"b6c9d2bc-60d6-494c-af57-574e85c4861c\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh" Oct 30 21:44:16.777160 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:16.776927 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6c9d2bc-60d6-494c-af57-574e85c4861c-workload-socket\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh\" (UID: \"b6c9d2bc-60d6-494c-af57-574e85c4861c\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh" Oct 30 21:44:16.777160 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:16.776951 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6c9d2bc-60d6-494c-af57-574e85c4861c-istio-podinfo\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh\" (UID: \"b6c9d2bc-60d6-494c-af57-574e85c4861c\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh" Oct 30 21:44:16.777160 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:16.776990 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6c9d2bc-60d6-494c-af57-574e85c4861c-credential-socket\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh\" (UID: \"b6c9d2bc-60d6-494c-af57-574e85c4861c\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh" Oct 30 21:44:16.777358 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:16.777332 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6c9d2bc-60d6-494c-af57-574e85c4861c-istio-data\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh\" (UID: \"b6c9d2bc-60d6-494c-af57-574e85c4861c\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh" Oct 30 21:44:16.777429 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:16.777378 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6c9d2bc-60d6-494c-af57-574e85c4861c-workload-socket\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh\" (UID: \"b6c9d2bc-60d6-494c-af57-574e85c4861c\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh" Oct 30 21:44:16.777469 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:16.777446 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6c9d2bc-60d6-494c-af57-574e85c4861c-workload-certs\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh\" (UID: \"b6c9d2bc-60d6-494c-af57-574e85c4861c\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh" Oct 30 21:44:16.777514 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:16.777485 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6c9d2bc-60d6-494c-af57-574e85c4861c-credential-socket\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh\" (UID: \"b6c9d2bc-60d6-494c-af57-574e85c4861c\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh" Oct 30 21:44:16.777694 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:16.777667 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6c9d2bc-60d6-494c-af57-574e85c4861c-istiod-ca-cert\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh\" (UID: \"b6c9d2bc-60d6-494c-af57-574e85c4861c\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh" Oct 30 21:44:16.779314 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:16.779284 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6c9d2bc-60d6-494c-af57-574e85c4861c-istio-envoy\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh\" (UID: \"b6c9d2bc-60d6-494c-af57-574e85c4861c\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh" Oct 30 21:44:16.779542 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:16.779522 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6c9d2bc-60d6-494c-af57-574e85c4861c-istio-podinfo\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh\" (UID: \"b6c9d2bc-60d6-494c-af57-574e85c4861c\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh" Oct 30 21:44:16.838926 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:16.838826 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6c9d2bc-60d6-494c-af57-574e85c4861c-istio-token\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh\" (UID: \"b6c9d2bc-60d6-494c-af57-574e85c4861c\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh" Oct 30 21:44:16.839330 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:16.839308 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqjhf\" (UniqueName: \"kubernetes.io/projected/b6c9d2bc-60d6-494c-af57-574e85c4861c-kube-api-access-tqjhf\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh\" (UID: \"b6c9d2bc-60d6-494c-af57-574e85c4861c\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh" Oct 30 21:44:16.875422 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:16.875387 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh" Oct 30 21:44:17.325313 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:17.325283 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh"] Oct 30 21:44:17.326223 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:44:17.326187 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6c9d2bc_60d6_494c_af57_574e85c4861c.slice/crio-7c3728c4fc5cc1f1d88636c20ecf018aa89ee9bb2963f4449e9212fc01dd3178 WatchSource:0}: Error finding container 7c3728c4fc5cc1f1d88636c20ecf018aa89ee9bb2963f4449e9212fc01dd3178: Status 404 returned error can't find the container with id 7c3728c4fc5cc1f1d88636c20ecf018aa89ee9bb2963f4449e9212fc01dd3178 Oct 30 21:44:17.328756 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:17.328722 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:44:17.328859 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:17.328799 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:44:17.328859 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:17.328847 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:44:17.355316 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:17.355287 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh" event={"ID":"b6c9d2bc-60d6-494c-af57-574e85c4861c","Type":"ContainerStarted","Data":"7c3728c4fc5cc1f1d88636c20ecf018aa89ee9bb2963f4449e9212fc01dd3178"} Oct 30 21:44:18.049394 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:18.049354 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c56d9ab2-j4wwt"] Oct 30 21:44:18.099290 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:18.099260 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c56d9ab2-j4wwt"] Oct 30 21:44:18.363588 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:18.363502 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh" event={"ID":"b6c9d2bc-60d6-494c-af57-574e85c4861c","Type":"ContainerStarted","Data":"a88051189ba8ed7703e8bcb3da5b8a155ae851a3fa5b21c33fe875d27363b58c"} Oct 30 21:44:18.449216 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:18.449176 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80c3b0cc-5eb0-4b50-88f3-6e37bbb7297e" path="/var/lib/kubelet/pods/80c3b0cc-5eb0-4b50-88f3-6e37bbb7297e/volumes" Oct 30 21:44:18.494251 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:18.494188 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh" podStartSLOduration=2.494170845 podStartE2EDuration="2.494170845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:18.492723674 +0000 UTC m=+1242.235244609" watchObservedRunningTime="2026-10-30 21:44:18.494170845 +0000 UTC m=+1242.236691781" Oct 30 21:44:18.876001 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:18.875966 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh" Oct 30 21:44:18.877796 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:18.877765 5748 patch_prober.go:29] interesting pod/e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 21:44:18.877940 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:18.877812 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh" podUID="b6c9d2bc-60d6-494c-af57-574e85c4861c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 21:44:19.876580 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:19.876539 5748 patch_prober.go:29] interesting pod/e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 21:44:19.877011 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:19.876630 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh" podUID="b6c9d2bc-60d6-494c-af57-574e85c4861c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 21:44:20.876556 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:20.876513 5748 patch_prober.go:29] interesting pod/e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 21:44:20.876724 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:20.876572 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh" podUID="b6c9d2bc-60d6-494c-af57-574e85c4861c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 21:44:21.876715 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:21.876682 5748 patch_prober.go:29] interesting pod/e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 21:44:21.877148 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:21.876733 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh" podUID="b6c9d2bc-60d6-494c-af57-574e85c4861c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 21:44:22.875756 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:22.875717 5748 patch_prober.go:29] interesting pod/e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 21:44:22.875960 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:22.875768 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh" podUID="b6c9d2bc-60d6-494c-af57-574e85c4861c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 21:44:23.052655 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:23.052619 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3399cba9-7rmq7"] Oct 30 21:44:23.127392 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:23.127296 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3399cba9-7rmq7"] Oct 30 21:44:23.876402 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:23.876358 5748 patch_prober.go:29] interesting pod/e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 21:44:23.876579 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:23.876421 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh" podUID="b6c9d2bc-60d6-494c-af57-574e85c4861c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 21:44:24.108940 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:24.107003 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-030f7de6-4t9x2"] Oct 30 21:44:24.119697 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:24.119662 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-030f7de6-4t9x2" Oct 30 21:44:24.167891 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:24.167856 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-030f7de6-4t9x2"] Oct 30 21:44:24.192082 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:24.192046 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee3cb1c0-6bfc-4983-9be4-98e1f9b0b762-maas-api-xks-ca\") pod \"maas-api-revoke-sub-030f7de6-4t9x2\" (UID: \"ee3cb1c0-6bfc-4983-9be4-98e1f9b0b762\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-030f7de6-4t9x2" Oct 30 21:44:24.192243 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:24.192111 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phh22\" (UniqueName: \"kubernetes.io/projected/ee3cb1c0-6bfc-4983-9be4-98e1f9b0b762-kube-api-access-phh22\") pod \"maas-api-revoke-sub-030f7de6-4t9x2\" (UID: \"ee3cb1c0-6bfc-4983-9be4-98e1f9b0b762\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-030f7de6-4t9x2" Oct 30 21:44:24.192307 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:24.192279 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee3cb1c0-6bfc-4983-9be4-98e1f9b0b762-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-030f7de6-4t9x2\" (UID: \"ee3cb1c0-6bfc-4983-9be4-98e1f9b0b762\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-030f7de6-4t9x2" Oct 30 21:44:24.214527 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:24.214493 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:44:24.214683 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:24.214493 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:44:24.214683 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:24.214513 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:44:24.214683 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:24.214616 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 21:44:24.293036 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:24.293006 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee3cb1c0-6bfc-4983-9be4-98e1f9b0b762-maas-api-xks-ca\") pod \"maas-api-revoke-sub-030f7de6-4t9x2\" (UID: \"ee3cb1c0-6bfc-4983-9be4-98e1f9b0b762\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-030f7de6-4t9x2" Oct 30 21:44:24.293205 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:24.293065 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phh22\" (UniqueName: \"kubernetes.io/projected/ee3cb1c0-6bfc-4983-9be4-98e1f9b0b762-kube-api-access-phh22\") pod \"maas-api-revoke-sub-030f7de6-4t9x2\" (UID: \"ee3cb1c0-6bfc-4983-9be4-98e1f9b0b762\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-030f7de6-4t9x2" Oct 30 21:44:24.293205 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:24.293109 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee3cb1c0-6bfc-4983-9be4-98e1f9b0b762-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-030f7de6-4t9x2\" (UID: \"ee3cb1c0-6bfc-4983-9be4-98e1f9b0b762\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-030f7de6-4t9x2" Oct 30 21:44:24.293806 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:24.293777 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee3cb1c0-6bfc-4983-9be4-98e1f9b0b762-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-030f7de6-4t9x2\" (UID: \"ee3cb1c0-6bfc-4983-9be4-98e1f9b0b762\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-030f7de6-4t9x2" Oct 30 21:44:24.295637 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:24.295607 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee3cb1c0-6bfc-4983-9be4-98e1f9b0b762-maas-api-xks-ca\") pod \"maas-api-revoke-sub-030f7de6-4t9x2\" (UID: \"ee3cb1c0-6bfc-4983-9be4-98e1f9b0b762\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-030f7de6-4t9x2" Oct 30 21:44:24.355276 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:24.355242 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phh22\" (UniqueName: \"kubernetes.io/projected/ee3cb1c0-6bfc-4983-9be4-98e1f9b0b762-kube-api-access-phh22\") pod \"maas-api-revoke-sub-030f7de6-4t9x2\" (UID: \"ee3cb1c0-6bfc-4983-9be4-98e1f9b0b762\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-030f7de6-4t9x2" Oct 30 21:44:24.418279 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:24.418225 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="384173e5-7d05-4873-915d-a333af9f35e0" path="/var/lib/kubelet/pods/384173e5-7d05-4873-915d-a333af9f35e0/volumes" Oct 30 21:44:24.434740 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:24.434709 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-030f7de6-4t9x2" Oct 30 21:44:24.820091 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:24.820061 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-030f7de6-4t9x2"] Oct 30 21:44:24.820292 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:44:24.820266 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee3cb1c0_6bfc_4983_9be4_98e1f9b0b762.slice/crio-236f7623179f31de8bef9a9e3d1ea24c5fc726410d770141ebfb15882a33a0e9 WatchSource:0}: Error finding container 236f7623179f31de8bef9a9e3d1ea24c5fc726410d770141ebfb15882a33a0e9: Status 404 returned error can't find the container with id 236f7623179f31de8bef9a9e3d1ea24c5fc726410d770141ebfb15882a33a0e9 Oct 30 21:44:24.880487 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:24.880455 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh" Oct 30 21:44:24.880760 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:24.880741 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh" Oct 30 21:44:24.881612 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:24.881595 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh" Oct 30 21:44:25.418281 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:25.418247 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-030f7de6-4t9x2" event={"ID":"ee3cb1c0-6bfc-4983-9be4-98e1f9b0b762","Type":"ContainerStarted","Data":"88aebcc1fb27c9a222b360aa7ef3eae00cc174ce0cdddd87da64548311e69ccf"} Oct 30 21:44:25.418689 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:25.418299 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-030f7de6-4t9x2" event={"ID":"ee3cb1c0-6bfc-4983-9be4-98e1f9b0b762","Type":"ContainerStarted","Data":"236f7623179f31de8bef9a9e3d1ea24c5fc726410d770141ebfb15882a33a0e9"} Oct 30 21:44:25.537284 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:25.537231 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-030f7de6-4t9x2" podStartSLOduration=1.537215971 podStartE2EDuration="1.537215971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:25.535963804 +0000 UTC m=+1249.278484739" watchObservedRunningTime="2026-10-30 21:44:25.537215971 +0000 UTC m=+1249.279736903" Oct 30 21:44:26.426540 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:26.426504 5748 generic.go:356] "Generic (PLEG): container finished" podID="ee3cb1c0-6bfc-4983-9be4-98e1f9b0b762" containerID="88aebcc1fb27c9a222b360aa7ef3eae00cc174ce0cdddd87da64548311e69ccf" exitCode=0 Oct 30 21:44:26.426964 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:26.426586 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-030f7de6-4t9x2" event={"ID":"ee3cb1c0-6bfc-4983-9be4-98e1f9b0b762","Type":"ContainerDied","Data":"88aebcc1fb27c9a222b360aa7ef3eae00cc174ce0cdddd87da64548311e69ccf"} Oct 30 21:44:27.024563 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:27.020426 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fd22cad-cskvk"] Oct 30 21:44:27.026820 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:27.026795 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd22cad-cskvk" Oct 30 21:44:27.078818 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:27.078771 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fd22cad-cskvk"] Oct 30 21:44:27.131118 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:27.131076 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da88fa3f-81d6-4fef-b231-7641ca76ea88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fd22cad-cskvk\" (UID: \"da88fa3f-81d6-4fef-b231-7641ca76ea88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd22cad-cskvk" Oct 30 21:44:27.131268 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:27.131152 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtdcj\" (UniqueName: \"kubernetes.io/projected/da88fa3f-81d6-4fef-b231-7641ca76ea88-kube-api-access-wtdcj\") pod \"maas-api-revoke-sub-1fd22cad-cskvk\" (UID: \"da88fa3f-81d6-4fef-b231-7641ca76ea88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd22cad-cskvk" Oct 30 21:44:27.131331 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:27.131306 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da88fa3f-81d6-4fef-b231-7641ca76ea88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fd22cad-cskvk\" (UID: \"da88fa3f-81d6-4fef-b231-7641ca76ea88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd22cad-cskvk" Oct 30 21:44:27.232421 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:27.232375 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da88fa3f-81d6-4fef-b231-7641ca76ea88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fd22cad-cskvk\" (UID: \"da88fa3f-81d6-4fef-b231-7641ca76ea88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd22cad-cskvk" Oct 30 21:44:27.232617 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:27.232559 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da88fa3f-81d6-4fef-b231-7641ca76ea88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fd22cad-cskvk\" (UID: \"da88fa3f-81d6-4fef-b231-7641ca76ea88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd22cad-cskvk" Oct 30 21:44:27.232689 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:27.232634 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtdcj\" (UniqueName: \"kubernetes.io/projected/da88fa3f-81d6-4fef-b231-7641ca76ea88-kube-api-access-wtdcj\") pod \"maas-api-revoke-sub-1fd22cad-cskvk\" (UID: \"da88fa3f-81d6-4fef-b231-7641ca76ea88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd22cad-cskvk" Oct 30 21:44:27.233157 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:27.233131 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da88fa3f-81d6-4fef-b231-7641ca76ea88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fd22cad-cskvk\" (UID: \"da88fa3f-81d6-4fef-b231-7641ca76ea88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd22cad-cskvk" Oct 30 21:44:27.235034 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:27.235008 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da88fa3f-81d6-4fef-b231-7641ca76ea88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fd22cad-cskvk\" (UID: \"da88fa3f-81d6-4fef-b231-7641ca76ea88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd22cad-cskvk" Oct 30 21:44:27.295092 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:27.295019 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtdcj\" (UniqueName: \"kubernetes.io/projected/da88fa3f-81d6-4fef-b231-7641ca76ea88-kube-api-access-wtdcj\") pod \"maas-api-revoke-sub-1fd22cad-cskvk\" (UID: \"da88fa3f-81d6-4fef-b231-7641ca76ea88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd22cad-cskvk" Oct 30 21:44:27.342284 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:27.342249 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd22cad-cskvk" Oct 30 21:44:27.587660 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:27.587633 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-030f7de6-4t9x2" Oct 30 21:44:27.726796 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:44:27.726765 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda88fa3f_81d6_4fef_b231_7641ca76ea88.slice/crio-ee6fcf41dca4025e7778275447593ea5f394177b1f69248e0601c57ca2015966 WatchSource:0}: Error finding container ee6fcf41dca4025e7778275447593ea5f394177b1f69248e0601c57ca2015966: Status 404 returned error can't find the container with id ee6fcf41dca4025e7778275447593ea5f394177b1f69248e0601c57ca2015966 Oct 30 21:44:27.738429 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:27.738399 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee3cb1c0-6bfc-4983-9be4-98e1f9b0b762-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee3cb1c0-6bfc-4983-9be4-98e1f9b0b762-maas-api-xks-ca\") pod \"ee3cb1c0-6bfc-4983-9be4-98e1f9b0b762\" (UID: \"ee3cb1c0-6bfc-4983-9be4-98e1f9b0b762\") " Oct 30 21:44:27.738579 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:27.738560 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee3cb1c0-6bfc-4983-9be4-98e1f9b0b762-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee3cb1c0-6bfc-4983-9be4-98e1f9b0b762-maas-api-openshift-ca\") pod \"ee3cb1c0-6bfc-4983-9be4-98e1f9b0b762\" (UID: \"ee3cb1c0-6bfc-4983-9be4-98e1f9b0b762\") " Oct 30 21:44:27.738687 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:27.738670 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee3cb1c0-6bfc-4983-9be4-98e1f9b0b762-kube-api-access-phh22\" (UniqueName: \"kubernetes.io/projected/ee3cb1c0-6bfc-4983-9be4-98e1f9b0b762-kube-api-access-phh22\") pod \"ee3cb1c0-6bfc-4983-9be4-98e1f9b0b762\" (UID: \"ee3cb1c0-6bfc-4983-9be4-98e1f9b0b762\") " Oct 30 21:44:27.739054 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:27.739027 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee3cb1c0-6bfc-4983-9be4-98e1f9b0b762-maas-api-openshift-ca" pod "ee3cb1c0-6bfc-4983-9be4-98e1f9b0b762" (UID: "ee3cb1c0-6bfc-4983-9be4-98e1f9b0b762"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:44:27.739352 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:27.739328 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee3cb1c0-6bfc-4983-9be4-98e1f9b0b762-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:27.740983 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:27.740952 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee3cb1c0-6bfc-4983-9be4-98e1f9b0b762-kube-api-access-phh22" pod "ee3cb1c0-6bfc-4983-9be4-98e1f9b0b762" (UID: "ee3cb1c0-6bfc-4983-9be4-98e1f9b0b762"). InnerVolumeSpecName "kube-api-access-phh22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:27.741094 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:27.741068 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee3cb1c0-6bfc-4983-9be4-98e1f9b0b762-maas-api-xks-ca" pod "ee3cb1c0-6bfc-4983-9be4-98e1f9b0b762" (UID: "ee3cb1c0-6bfc-4983-9be4-98e1f9b0b762"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:44:27.763717 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:27.763692 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fd22cad-cskvk"] Oct 30 21:44:27.840237 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:27.840148 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee3cb1c0-6bfc-4983-9be4-98e1f9b0b762-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:27.840237 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:27.840181 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phh22\" (UniqueName: \"kubernetes.io/projected/ee3cb1c0-6bfc-4983-9be4-98e1f9b0b762-kube-api-access-phh22\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:28.048867 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:28.048823 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b425e42-2fnd9"] Oct 30 21:44:28.053759 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:28.053719 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b425e42-2fnd9"] Oct 30 21:44:28.418695 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:28.418659 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b6e60cc-ea7d-4c1e-bef8-3178637efa0b" path="/var/lib/kubelet/pods/6b6e60cc-ea7d-4c1e-bef8-3178637efa0b/volumes" Oct 30 21:44:28.443847 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:28.443820 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-030f7de6-4t9x2" Oct 30 21:44:28.443847 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:28.443813 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-030f7de6-4t9x2" event={"ID":"ee3cb1c0-6bfc-4983-9be4-98e1f9b0b762","Type":"ContainerDied","Data":"236f7623179f31de8bef9a9e3d1ea24c5fc726410d770141ebfb15882a33a0e9"} Oct 30 21:44:28.444124 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:28.443877 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="236f7623179f31de8bef9a9e3d1ea24c5fc726410d770141ebfb15882a33a0e9" Oct 30 21:44:28.445662 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:28.445638 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd22cad-cskvk" event={"ID":"da88fa3f-81d6-4fef-b231-7641ca76ea88","Type":"ContainerStarted","Data":"86875bb2bc2a7c198275c70b35b05c42879c86f7f9ea7cf907d9f403e31b1027"} Oct 30 21:44:28.445792 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:28.445687 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd22cad-cskvk" event={"ID":"da88fa3f-81d6-4fef-b231-7641ca76ea88","Type":"ContainerStarted","Data":"ee6fcf41dca4025e7778275447593ea5f394177b1f69248e0601c57ca2015966"} Oct 30 21:44:28.568013 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:28.567963 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd22cad-cskvk" podStartSLOduration=2.5679489159999997 podStartE2EDuration="2.567948916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:28.565218673 +0000 UTC m=+1252.307739609" watchObservedRunningTime="2026-10-30 21:44:28.567948916 +0000 UTC m=+1252.310469851" Oct 30 21:44:29.050365 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:29.050321 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd95a157-qmjvl"] Oct 30 21:44:29.053509 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:29.053484 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd95a157-qmjvl"] Oct 30 21:44:29.455107 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:29.455072 5748 generic.go:356] "Generic (PLEG): container finished" podID="da88fa3f-81d6-4fef-b231-7641ca76ea88" containerID="86875bb2bc2a7c198275c70b35b05c42879c86f7f9ea7cf907d9f403e31b1027" exitCode=0 Oct 30 21:44:29.455299 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:29.455134 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd22cad-cskvk" event={"ID":"da88fa3f-81d6-4fef-b231-7641ca76ea88","Type":"ContainerDied","Data":"86875bb2bc2a7c198275c70b35b05c42879c86f7f9ea7cf907d9f403e31b1027"} Oct 30 21:44:30.148511 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:44:30.148477 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:44:30.418764 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:30.418683 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1acf0f40-6f96-4914-8350-c5e19015968c" path="/var/lib/kubelet/pods/1acf0f40-6f96-4914-8350-c5e19015968c/volumes" Oct 30 21:44:30.610390 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:30.610364 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd22cad-cskvk" Oct 30 21:44:30.777512 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:30.777414 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da88fa3f-81d6-4fef-b231-7641ca76ea88-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da88fa3f-81d6-4fef-b231-7641ca76ea88-maas-api-xks-ca\") pod \"da88fa3f-81d6-4fef-b231-7641ca76ea88\" (UID: \"da88fa3f-81d6-4fef-b231-7641ca76ea88\") " Oct 30 21:44:30.777690 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:30.777556 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da88fa3f-81d6-4fef-b231-7641ca76ea88-kube-api-access-wtdcj\" (UniqueName: \"kubernetes.io/projected/da88fa3f-81d6-4fef-b231-7641ca76ea88-kube-api-access-wtdcj\") pod \"da88fa3f-81d6-4fef-b231-7641ca76ea88\" (UID: \"da88fa3f-81d6-4fef-b231-7641ca76ea88\") " Oct 30 21:44:30.777690 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:30.777630 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da88fa3f-81d6-4fef-b231-7641ca76ea88-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da88fa3f-81d6-4fef-b231-7641ca76ea88-maas-api-openshift-ca\") pod \"da88fa3f-81d6-4fef-b231-7641ca76ea88\" (UID: \"da88fa3f-81d6-4fef-b231-7641ca76ea88\") " Oct 30 21:44:30.777999 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:30.777975 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da88fa3f-81d6-4fef-b231-7641ca76ea88-maas-api-openshift-ca" pod "da88fa3f-81d6-4fef-b231-7641ca76ea88" (UID: "da88fa3f-81d6-4fef-b231-7641ca76ea88"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:44:30.778107 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:30.778092 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da88fa3f-81d6-4fef-b231-7641ca76ea88-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:30.779579 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:30.779552 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da88fa3f-81d6-4fef-b231-7641ca76ea88-kube-api-access-wtdcj" pod "da88fa3f-81d6-4fef-b231-7641ca76ea88" (UID: "da88fa3f-81d6-4fef-b231-7641ca76ea88"). InnerVolumeSpecName "kube-api-access-wtdcj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:30.779686 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:30.779577 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da88fa3f-81d6-4fef-b231-7641ca76ea88-maas-api-xks-ca" pod "da88fa3f-81d6-4fef-b231-7641ca76ea88" (UID: "da88fa3f-81d6-4fef-b231-7641ca76ea88"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:44:30.879646 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:30.879610 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da88fa3f-81d6-4fef-b231-7641ca76ea88-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:30.879646 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:30.879640 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtdcj\" (UniqueName: \"kubernetes.io/projected/da88fa3f-81d6-4fef-b231-7641ca76ea88-kube-api-access-wtdcj\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:31.472871 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:31.472839 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd22cad-cskvk" Oct 30 21:44:31.473272 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:31.472839 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd22cad-cskvk" event={"ID":"da88fa3f-81d6-4fef-b231-7641ca76ea88","Type":"ContainerDied","Data":"ee6fcf41dca4025e7778275447593ea5f394177b1f69248e0601c57ca2015966"} Oct 30 21:44:31.473272 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:31.472960 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee6fcf41dca4025e7778275447593ea5f394177b1f69248e0601c57ca2015966" Oct 30 21:44:36.575327 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:36.575294 5748 scope.go:122] "RemoveContainer" containerID="51a5f8993a72b75d95dd38093caadd19c59a53a10e4d1da3d3b518b54f6da0d5" Oct 30 21:44:37.050430 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:37.050397 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5758a8e8-2s9zv"] Oct 30 21:44:37.100456 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:37.100415 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5758a8e8-2s9zv"] Oct 30 21:44:38.418550 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:38.418519 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd1c21b8-d727-4ab8-9260-69a7fb3340c0" path="/var/lib/kubelet/pods/fd1c21b8-d727-4ab8-9260-69a7fb3340c0/volumes" Oct 30 21:44:40.745548 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:40.745513 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83bee69b-khpsw"] Oct 30 21:44:40.746083 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:40.746069 5748 cpu_manager.go:414] "containerMap: removing container" podUID="da88fa3f-81d6-4fef-b231-7641ca76ea88" containerName="revoke-keys" Oct 30 21:44:40.746138 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:40.746085 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da88fa3f-81d6-4fef-b231-7641ca76ea88" containerName="revoke-keys" Oct 30 21:44:40.746138 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:40.746110 5748 cpu_manager.go:414] "containerMap: removing container" podUID="ee3cb1c0-6bfc-4983-9be4-98e1f9b0b762" containerName="revoke-keys" Oct 30 21:44:40.746138 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:40.746116 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee3cb1c0-6bfc-4983-9be4-98e1f9b0b762" containerName="revoke-keys" Oct 30 21:44:40.746236 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:40.746224 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="da88fa3f-81d6-4fef-b231-7641ca76ea88" containerName="revoke-keys" Oct 30 21:44:40.746236 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:40.746236 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee3cb1c0-6bfc-4983-9be4-98e1f9b0b762" containerName="revoke-keys" Oct 30 21:44:40.751515 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:40.751497 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bee69b-khpsw" Oct 30 21:44:40.796738 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:40.796708 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af33c160-83ca-4ac8-b2fd-4273ff2c0656-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83bee69b-khpsw\" (UID: \"af33c160-83ca-4ac8-b2fd-4273ff2c0656\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bee69b-khpsw" Oct 30 21:44:40.796875 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:40.796762 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af33c160-83ca-4ac8-b2fd-4273ff2c0656-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83bee69b-khpsw\" (UID: \"af33c160-83ca-4ac8-b2fd-4273ff2c0656\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bee69b-khpsw" Oct 30 21:44:40.796966 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:40.796887 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x46z6\" (UniqueName: \"kubernetes.io/projected/af33c160-83ca-4ac8-b2fd-4273ff2c0656-kube-api-access-x46z6\") pod \"maas-api-revoke-sub-83bee69b-khpsw\" (UID: \"af33c160-83ca-4ac8-b2fd-4273ff2c0656\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bee69b-khpsw" Oct 30 21:44:40.808142 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:40.808118 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83bee69b-khpsw"] Oct 30 21:44:40.854290 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:40.854261 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:44:40.854475 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:40.854315 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:44:40.854475 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:40.854358 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 21:44:40.855482 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:40.855463 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:44:40.898367 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:40.898331 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af33c160-83ca-4ac8-b2fd-4273ff2c0656-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83bee69b-khpsw\" (UID: \"af33c160-83ca-4ac8-b2fd-4273ff2c0656\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bee69b-khpsw" Oct 30 21:44:40.898551 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:40.898391 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af33c160-83ca-4ac8-b2fd-4273ff2c0656-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83bee69b-khpsw\" (UID: \"af33c160-83ca-4ac8-b2fd-4273ff2c0656\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bee69b-khpsw" Oct 30 21:44:40.898551 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:40.898426 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x46z6\" (UniqueName: \"kubernetes.io/projected/af33c160-83ca-4ac8-b2fd-4273ff2c0656-kube-api-access-x46z6\") pod \"maas-api-revoke-sub-83bee69b-khpsw\" (UID: \"af33c160-83ca-4ac8-b2fd-4273ff2c0656\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bee69b-khpsw" Oct 30 21:44:40.899089 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:40.899063 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af33c160-83ca-4ac8-b2fd-4273ff2c0656-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83bee69b-khpsw\" (UID: \"af33c160-83ca-4ac8-b2fd-4273ff2c0656\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bee69b-khpsw" Oct 30 21:44:40.900871 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:40.900853 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af33c160-83ca-4ac8-b2fd-4273ff2c0656-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83bee69b-khpsw\" (UID: \"af33c160-83ca-4ac8-b2fd-4273ff2c0656\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bee69b-khpsw" Oct 30 21:44:40.960193 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:40.960167 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x46z6\" (UniqueName: \"kubernetes.io/projected/af33c160-83ca-4ac8-b2fd-4273ff2c0656-kube-api-access-x46z6\") pod \"maas-api-revoke-sub-83bee69b-khpsw\" (UID: \"af33c160-83ca-4ac8-b2fd-4273ff2c0656\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bee69b-khpsw" Oct 30 21:44:41.064921 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:41.064821 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bee69b-khpsw" Oct 30 21:44:41.446528 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:41.446485 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83bee69b-khpsw"] Oct 30 21:44:41.454765 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:44:41.454719 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf33c160_83ca_4ac8_b2fd_4273ff2c0656.slice/crio-0ed41616e6506ecfed5040ddadabb2ce5492e5adacaa5fddc89493af44858ee6 WatchSource:0}: Error finding container 0ed41616e6506ecfed5040ddadabb2ce5492e5adacaa5fddc89493af44858ee6: Status 404 returned error can't find the container with id 0ed41616e6506ecfed5040ddadabb2ce5492e5adacaa5fddc89493af44858ee6 Oct 30 21:44:41.548417 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:41.548382 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bee69b-khpsw" event={"ID":"af33c160-83ca-4ac8-b2fd-4273ff2c0656","Type":"ContainerStarted","Data":"0ed41616e6506ecfed5040ddadabb2ce5492e5adacaa5fddc89493af44858ee6"} Oct 30 21:44:42.557161 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:42.557128 5748 generic.go:356] "Generic (PLEG): container finished" podID="af33c160-83ca-4ac8-b2fd-4273ff2c0656" containerID="e763bf5a728913cf6824ab8015eeced7e1ea6fd290eab77f832ff20fde392ba7" exitCode=0 Oct 30 21:44:42.557576 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:42.557213 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bee69b-khpsw" event={"ID":"af33c160-83ca-4ac8-b2fd-4273ff2c0656","Type":"ContainerDied","Data":"e763bf5a728913cf6824ab8015eeced7e1ea6fd290eab77f832ff20fde392ba7"} Oct 30 21:44:42.727464 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:42.727435 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8192d23-tm9fg"] Oct 30 21:44:42.731573 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:42.731553 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8192d23-tm9fg" Oct 30 21:44:42.790225 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:42.790192 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8192d23-tm9fg"] Oct 30 21:44:42.823079 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:42.823046 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50a27536-551f-47f3-8ae8-d754fadf7a71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8192d23-tm9fg\" (UID: \"50a27536-551f-47f3-8ae8-d754fadf7a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8192d23-tm9fg" Oct 30 21:44:42.823262 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:42.823233 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50a27536-551f-47f3-8ae8-d754fadf7a71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8192d23-tm9fg\" (UID: \"50a27536-551f-47f3-8ae8-d754fadf7a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8192d23-tm9fg" Oct 30 21:44:42.823330 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:42.823280 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhqqn\" (UniqueName: \"kubernetes.io/projected/50a27536-551f-47f3-8ae8-d754fadf7a71-kube-api-access-hhqqn\") pod \"maas-api-revoke-sub-f8192d23-tm9fg\" (UID: \"50a27536-551f-47f3-8ae8-d754fadf7a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8192d23-tm9fg" Oct 30 21:44:42.924689 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:42.924658 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50a27536-551f-47f3-8ae8-d754fadf7a71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8192d23-tm9fg\" (UID: \"50a27536-551f-47f3-8ae8-d754fadf7a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8192d23-tm9fg" Oct 30 21:44:42.924886 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:42.924706 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhqqn\" (UniqueName: \"kubernetes.io/projected/50a27536-551f-47f3-8ae8-d754fadf7a71-kube-api-access-hhqqn\") pod \"maas-api-revoke-sub-f8192d23-tm9fg\" (UID: \"50a27536-551f-47f3-8ae8-d754fadf7a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8192d23-tm9fg" Oct 30 21:44:42.924886 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:42.924852 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50a27536-551f-47f3-8ae8-d754fadf7a71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8192d23-tm9fg\" (UID: \"50a27536-551f-47f3-8ae8-d754fadf7a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8192d23-tm9fg" Oct 30 21:44:42.925370 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:42.925351 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50a27536-551f-47f3-8ae8-d754fadf7a71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8192d23-tm9fg\" (UID: \"50a27536-551f-47f3-8ae8-d754fadf7a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8192d23-tm9fg" Oct 30 21:44:42.927095 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:42.927073 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50a27536-551f-47f3-8ae8-d754fadf7a71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8192d23-tm9fg\" (UID: \"50a27536-551f-47f3-8ae8-d754fadf7a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8192d23-tm9fg" Oct 30 21:44:42.986481 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:42.986398 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhqqn\" (UniqueName: \"kubernetes.io/projected/50a27536-551f-47f3-8ae8-d754fadf7a71-kube-api-access-hhqqn\") pod \"maas-api-revoke-sub-f8192d23-tm9fg\" (UID: \"50a27536-551f-47f3-8ae8-d754fadf7a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8192d23-tm9fg" Oct 30 21:44:43.045442 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:43.045406 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8192d23-tm9fg" Oct 30 21:44:43.424831 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:43.424801 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8192d23-tm9fg"] Oct 30 21:44:43.425545 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:44:43.425519 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50a27536_551f_47f3_8ae8_d754fadf7a71.slice/crio-00378a58fbed426947d2b1abcdb8262a4e1b028a2fdcc6e73692c042c2670ac9 WatchSource:0}: Error finding container 00378a58fbed426947d2b1abcdb8262a4e1b028a2fdcc6e73692c042c2670ac9: Status 404 returned error can't find the container with id 00378a58fbed426947d2b1abcdb8262a4e1b028a2fdcc6e73692c042c2670ac9 Oct 30 21:44:43.580107 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:43.580072 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8192d23-tm9fg" event={"ID":"50a27536-551f-47f3-8ae8-d754fadf7a71","Type":"ContainerStarted","Data":"00378a58fbed426947d2b1abcdb8262a4e1b028a2fdcc6e73692c042c2670ac9"} Oct 30 21:44:43.725678 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:43.725651 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bee69b-khpsw" Oct 30 21:44:43.855057 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:43.853207 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af33c160-83ca-4ac8-b2fd-4273ff2c0656-kube-api-access-x46z6\" (UniqueName: \"kubernetes.io/projected/af33c160-83ca-4ac8-b2fd-4273ff2c0656-kube-api-access-x46z6\") pod \"af33c160-83ca-4ac8-b2fd-4273ff2c0656\" (UID: \"af33c160-83ca-4ac8-b2fd-4273ff2c0656\") " Oct 30 21:44:43.855057 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:43.853428 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af33c160-83ca-4ac8-b2fd-4273ff2c0656-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af33c160-83ca-4ac8-b2fd-4273ff2c0656-maas-api-openshift-ca\") pod \"af33c160-83ca-4ac8-b2fd-4273ff2c0656\" (UID: \"af33c160-83ca-4ac8-b2fd-4273ff2c0656\") " Oct 30 21:44:43.855057 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:43.853564 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af33c160-83ca-4ac8-b2fd-4273ff2c0656-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af33c160-83ca-4ac8-b2fd-4273ff2c0656-maas-api-xks-ca\") pod \"af33c160-83ca-4ac8-b2fd-4273ff2c0656\" (UID: \"af33c160-83ca-4ac8-b2fd-4273ff2c0656\") " Oct 30 21:44:43.855057 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:43.855040 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af33c160-83ca-4ac8-b2fd-4273ff2c0656-maas-api-openshift-ca" pod "af33c160-83ca-4ac8-b2fd-4273ff2c0656" (UID: "af33c160-83ca-4ac8-b2fd-4273ff2c0656"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:44:43.857657 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:43.857628 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af33c160-83ca-4ac8-b2fd-4273ff2c0656-maas-api-xks-ca" pod "af33c160-83ca-4ac8-b2fd-4273ff2c0656" (UID: "af33c160-83ca-4ac8-b2fd-4273ff2c0656"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:44:43.859319 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:43.859290 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af33c160-83ca-4ac8-b2fd-4273ff2c0656-kube-api-access-x46z6" pod "af33c160-83ca-4ac8-b2fd-4273ff2c0656" (UID: "af33c160-83ca-4ac8-b2fd-4273ff2c0656"). InnerVolumeSpecName "kube-api-access-x46z6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:43.955753 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:43.955719 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af33c160-83ca-4ac8-b2fd-4273ff2c0656-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:43.955753 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:43.955749 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x46z6\" (UniqueName: \"kubernetes.io/projected/af33c160-83ca-4ac8-b2fd-4273ff2c0656-kube-api-access-x46z6\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:43.955753 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:43.955760 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af33c160-83ca-4ac8-b2fd-4273ff2c0656-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:44.594249 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:44.594219 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bee69b-khpsw" Oct 30 21:44:44.594638 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:44.594245 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bee69b-khpsw" event={"ID":"af33c160-83ca-4ac8-b2fd-4273ff2c0656","Type":"ContainerDied","Data":"0ed41616e6506ecfed5040ddadabb2ce5492e5adacaa5fddc89493af44858ee6"} Oct 30 21:44:44.594638 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:44.594299 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ed41616e6506ecfed5040ddadabb2ce5492e5adacaa5fddc89493af44858ee6" Oct 30 21:44:44.595921 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:44.595873 5748 generic.go:356] "Generic (PLEG): container finished" podID="50a27536-551f-47f3-8ae8-d754fadf7a71" containerID="13d95474aaedca653593a2628ab64c3890827da0cf640eb0db75c537a7177bc1" exitCode=0 Oct 30 21:44:44.596040 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:44.595957 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8192d23-tm9fg" event={"ID":"50a27536-551f-47f3-8ae8-d754fadf7a71","Type":"ContainerDied","Data":"13d95474aaedca653593a2628ab64c3890827da0cf640eb0db75c537a7177bc1"} Oct 30 21:44:45.764371 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:45.764344 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8192d23-tm9fg" Oct 30 21:44:45.892887 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:45.892855 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50a27536-551f-47f3-8ae8-d754fadf7a71-kube-api-access-hhqqn\" (UniqueName: \"kubernetes.io/projected/50a27536-551f-47f3-8ae8-d754fadf7a71-kube-api-access-hhqqn\") pod \"50a27536-551f-47f3-8ae8-d754fadf7a71\" (UID: \"50a27536-551f-47f3-8ae8-d754fadf7a71\") " Oct 30 21:44:45.893148 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:45.893129 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50a27536-551f-47f3-8ae8-d754fadf7a71-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50a27536-551f-47f3-8ae8-d754fadf7a71-maas-api-xks-ca\") pod \"50a27536-551f-47f3-8ae8-d754fadf7a71\" (UID: \"50a27536-551f-47f3-8ae8-d754fadf7a71\") " Oct 30 21:44:45.893219 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:45.893173 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50a27536-551f-47f3-8ae8-d754fadf7a71-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50a27536-551f-47f3-8ae8-d754fadf7a71-maas-api-openshift-ca\") pod \"50a27536-551f-47f3-8ae8-d754fadf7a71\" (UID: \"50a27536-551f-47f3-8ae8-d754fadf7a71\") " Oct 30 21:44:45.893926 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:45.893879 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50a27536-551f-47f3-8ae8-d754fadf7a71-maas-api-openshift-ca" pod "50a27536-551f-47f3-8ae8-d754fadf7a71" (UID: "50a27536-551f-47f3-8ae8-d754fadf7a71"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:44:45.895459 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:45.895434 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50a27536-551f-47f3-8ae8-d754fadf7a71-kube-api-access-hhqqn" pod "50a27536-551f-47f3-8ae8-d754fadf7a71" (UID: "50a27536-551f-47f3-8ae8-d754fadf7a71"). InnerVolumeSpecName "kube-api-access-hhqqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:45.895646 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:45.895624 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50a27536-551f-47f3-8ae8-d754fadf7a71-maas-api-xks-ca" pod "50a27536-551f-47f3-8ae8-d754fadf7a71" (UID: "50a27536-551f-47f3-8ae8-d754fadf7a71"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:44:45.995162 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:45.995127 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50a27536-551f-47f3-8ae8-d754fadf7a71-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:45.995162 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:45.995161 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50a27536-551f-47f3-8ae8-d754fadf7a71-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:45.995371 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:45.995176 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhqqn\" (UniqueName: \"kubernetes.io/projected/50a27536-551f-47f3-8ae8-d754fadf7a71-kube-api-access-hhqqn\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:46.611660 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:46.611628 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8192d23-tm9fg" Oct 30 21:44:46.611835 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:46.611621 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8192d23-tm9fg" event={"ID":"50a27536-551f-47f3-8ae8-d754fadf7a71","Type":"ContainerDied","Data":"00378a58fbed426947d2b1abcdb8262a4e1b028a2fdcc6e73692c042c2670ac9"} Oct 30 21:44:46.611835 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:46.611754 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00378a58fbed426947d2b1abcdb8262a4e1b028a2fdcc6e73692c042c2670ac9" Oct 30 21:44:47.049719 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:47.049683 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85b4ac7a-g9rn7"] Oct 30 21:44:47.053406 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:47.053373 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85b4ac7a-g9rn7"] Oct 30 21:44:48.419209 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:48.419172 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="536af565-9de6-4203-b850-39ec9ab23315" path="/var/lib/kubelet/pods/536af565-9de6-4203-b850-39ec9ab23315/volumes" Oct 30 21:44:49.058180 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:49.058148 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-796e1be4-cf4mn"] Oct 30 21:44:49.108710 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:49.108679 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-796e1be4-cf4mn"] Oct 30 21:44:50.419855 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:50.419808 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad271c61-4a3c-4432-baed-2e4f5045c471" path="/var/lib/kubelet/pods/ad271c61-4a3c-4432-baed-2e4f5045c471/volumes" Oct 30 21:44:51.200111 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.200072 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8babaae-hl6tw"] Oct 30 21:44:51.200672 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.200656 5748 cpu_manager.go:414] "containerMap: removing container" podUID="50a27536-551f-47f3-8ae8-d754fadf7a71" containerName="revoke-keys" Oct 30 21:44:51.200716 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.200674 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50a27536-551f-47f3-8ae8-d754fadf7a71" containerName="revoke-keys" Oct 30 21:44:51.200716 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.200686 5748 cpu_manager.go:414] "containerMap: removing container" podUID="af33c160-83ca-4ac8-b2fd-4273ff2c0656" containerName="revoke-keys" Oct 30 21:44:51.200716 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.200691 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af33c160-83ca-4ac8-b2fd-4273ff2c0656" containerName="revoke-keys" Oct 30 21:44:51.200815 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.200792 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="50a27536-551f-47f3-8ae8-d754fadf7a71" containerName="revoke-keys" Oct 30 21:44:51.200815 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.200803 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="af33c160-83ca-4ac8-b2fd-4273ff2c0656" containerName="revoke-keys" Oct 30 21:44:51.204184 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.204165 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8babaae-hl6tw" Oct 30 21:44:51.258625 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.258592 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:44:51.258843 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.258715 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:44:51.309315 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.309280 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 21:44:51.309513 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.309315 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:44:51.313070 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.313041 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8babaae-hl6tw"] Oct 30 21:44:51.316027 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.316005 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02b9f950-cq6mz"] Oct 30 21:44:51.320163 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.320148 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b9f950-cq6mz" Oct 30 21:44:51.375137 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.375036 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/889613b9-6b8d-4ebc-a9ac-d9b2e9d8d817-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8babaae-hl6tw\" (UID: \"889613b9-6b8d-4ebc-a9ac-d9b2e9d8d817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8babaae-hl6tw" Oct 30 21:44:51.375872 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.375836 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/889613b9-6b8d-4ebc-a9ac-d9b2e9d8d817-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8babaae-hl6tw\" (UID: \"889613b9-6b8d-4ebc-a9ac-d9b2e9d8d817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8babaae-hl6tw" Oct 30 21:44:51.376040 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.375972 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrgpb\" (UniqueName: \"kubernetes.io/projected/889613b9-6b8d-4ebc-a9ac-d9b2e9d8d817-kube-api-access-zrgpb\") pod \"maas-api-revoke-sub-b8babaae-hl6tw\" (UID: \"889613b9-6b8d-4ebc-a9ac-d9b2e9d8d817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8babaae-hl6tw" Oct 30 21:44:51.381382 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.381358 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02b9f950-cq6mz"] Oct 30 21:44:51.477389 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.477283 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4b02ecf-7f32-4555-a5b0-2821f21ad666-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02b9f950-cq6mz\" (UID: \"d4b02ecf-7f32-4555-a5b0-2821f21ad666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b9f950-cq6mz" Oct 30 21:44:51.477389 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.477372 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/889613b9-6b8d-4ebc-a9ac-d9b2e9d8d817-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8babaae-hl6tw\" (UID: \"889613b9-6b8d-4ebc-a9ac-d9b2e9d8d817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8babaae-hl6tw" Oct 30 21:44:51.477910 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.477422 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrgpb\" (UniqueName: \"kubernetes.io/projected/889613b9-6b8d-4ebc-a9ac-d9b2e9d8d817-kube-api-access-zrgpb\") pod \"maas-api-revoke-sub-b8babaae-hl6tw\" (UID: \"889613b9-6b8d-4ebc-a9ac-d9b2e9d8d817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8babaae-hl6tw" Oct 30 21:44:51.477910 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.477629 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfksk\" (UniqueName: \"kubernetes.io/projected/d4b02ecf-7f32-4555-a5b0-2821f21ad666-kube-api-access-lfksk\") pod \"maas-api-revoke-sub-02b9f950-cq6mz\" (UID: \"d4b02ecf-7f32-4555-a5b0-2821f21ad666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b9f950-cq6mz" Oct 30 21:44:51.477910 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.477792 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/889613b9-6b8d-4ebc-a9ac-d9b2e9d8d817-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8babaae-hl6tw\" (UID: \"889613b9-6b8d-4ebc-a9ac-d9b2e9d8d817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8babaae-hl6tw" Oct 30 21:44:51.477910 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.477854 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4b02ecf-7f32-4555-a5b0-2821f21ad666-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02b9f950-cq6mz\" (UID: \"d4b02ecf-7f32-4555-a5b0-2821f21ad666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b9f950-cq6mz" Oct 30 21:44:51.478283 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.478265 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/889613b9-6b8d-4ebc-a9ac-d9b2e9d8d817-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8babaae-hl6tw\" (UID: \"889613b9-6b8d-4ebc-a9ac-d9b2e9d8d817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8babaae-hl6tw" Oct 30 21:44:51.480146 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.480128 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/889613b9-6b8d-4ebc-a9ac-d9b2e9d8d817-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8babaae-hl6tw\" (UID: \"889613b9-6b8d-4ebc-a9ac-d9b2e9d8d817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8babaae-hl6tw" Oct 30 21:44:51.539096 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.539062 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrgpb\" (UniqueName: \"kubernetes.io/projected/889613b9-6b8d-4ebc-a9ac-d9b2e9d8d817-kube-api-access-zrgpb\") pod \"maas-api-revoke-sub-b8babaae-hl6tw\" (UID: \"889613b9-6b8d-4ebc-a9ac-d9b2e9d8d817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8babaae-hl6tw" Oct 30 21:44:51.579224 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.579186 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4b02ecf-7f32-4555-a5b0-2821f21ad666-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02b9f950-cq6mz\" (UID: \"d4b02ecf-7f32-4555-a5b0-2821f21ad666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b9f950-cq6mz" Oct 30 21:44:51.579401 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.579343 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfksk\" (UniqueName: \"kubernetes.io/projected/d4b02ecf-7f32-4555-a5b0-2821f21ad666-kube-api-access-lfksk\") pod \"maas-api-revoke-sub-02b9f950-cq6mz\" (UID: \"d4b02ecf-7f32-4555-a5b0-2821f21ad666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b9f950-cq6mz" Oct 30 21:44:51.579451 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.579433 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4b02ecf-7f32-4555-a5b0-2821f21ad666-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02b9f950-cq6mz\" (UID: \"d4b02ecf-7f32-4555-a5b0-2821f21ad666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b9f950-cq6mz" Oct 30 21:44:51.579877 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.579850 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4b02ecf-7f32-4555-a5b0-2821f21ad666-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02b9f950-cq6mz\" (UID: \"d4b02ecf-7f32-4555-a5b0-2821f21ad666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b9f950-cq6mz" Oct 30 21:44:51.581708 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.581684 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4b02ecf-7f32-4555-a5b0-2821f21ad666-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02b9f950-cq6mz\" (UID: \"d4b02ecf-7f32-4555-a5b0-2821f21ad666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b9f950-cq6mz" Oct 30 21:44:51.612824 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.612789 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r"] Oct 30 21:44:51.621836 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.621801 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r" Oct 30 21:44:51.643536 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.643507 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfksk\" (UniqueName: \"kubernetes.io/projected/d4b02ecf-7f32-4555-a5b0-2821f21ad666-kube-api-access-lfksk\") pod \"maas-api-revoke-sub-02b9f950-cq6mz\" (UID: \"d4b02ecf-7f32-4555-a5b0-2821f21ad666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b9f950-cq6mz" Oct 30 21:44:51.649033 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.649008 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4"] Oct 30 21:44:51.653824 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.653795 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4" Oct 30 21:44:51.678332 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.678297 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r"] Oct 30 21:44:51.680799 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.680767 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d063b03-bc00-48bf-a91c-dfb31c286788-istio-podinfo\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4\" (UID: \"2d063b03-bc00-48bf-a91c-dfb31c286788\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4" Oct 30 21:44:51.680961 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.680811 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-istiod-ca-cert\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r\" (UID: \"ca6f3e2f-0392-4d98-a47f-65deafaf93fb\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r" Oct 30 21:44:51.680961 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.680859 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-istio-envoy\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r\" (UID: \"ca6f3e2f-0392-4d98-a47f-65deafaf93fb\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r" Oct 30 21:44:51.680961 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.680887 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-istio-token\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r\" (UID: \"ca6f3e2f-0392-4d98-a47f-65deafaf93fb\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r" Oct 30 21:44:51.680961 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.680945 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d063b03-bc00-48bf-a91c-dfb31c286788-istiod-ca-cert\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4\" (UID: \"2d063b03-bc00-48bf-a91c-dfb31c286788\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4" Oct 30 21:44:51.681181 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.680996 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfw65\" (UniqueName: \"kubernetes.io/projected/2d063b03-bc00-48bf-a91c-dfb31c286788-kube-api-access-cfw65\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4\" (UID: \"2d063b03-bc00-48bf-a91c-dfb31c286788\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4" Oct 30 21:44:51.681181 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.681029 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-workload-certs\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r\" (UID: \"ca6f3e2f-0392-4d98-a47f-65deafaf93fb\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r" Oct 30 21:44:51.681181 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.681052 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-istio-podinfo\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r\" (UID: \"ca6f3e2f-0392-4d98-a47f-65deafaf93fb\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r" Oct 30 21:44:51.681181 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.681112 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d063b03-bc00-48bf-a91c-dfb31c286788-credential-socket\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4\" (UID: \"2d063b03-bc00-48bf-a91c-dfb31c286788\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4" Oct 30 21:44:51.681181 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.681138 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d063b03-bc00-48bf-a91c-dfb31c286788-istio-token\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4\" (UID: \"2d063b03-bc00-48bf-a91c-dfb31c286788\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4" Oct 30 21:44:51.681426 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.681201 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-istio-data\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r\" (UID: \"ca6f3e2f-0392-4d98-a47f-65deafaf93fb\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r" Oct 30 21:44:51.681426 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.681247 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d063b03-bc00-48bf-a91c-dfb31c286788-workload-socket\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4\" (UID: \"2d063b03-bc00-48bf-a91c-dfb31c286788\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4" Oct 30 21:44:51.681426 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.681292 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d063b03-bc00-48bf-a91c-dfb31c286788-istio-envoy\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4\" (UID: \"2d063b03-bc00-48bf-a91c-dfb31c286788\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4" Oct 30 21:44:51.681426 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.681318 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-credential-socket\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r\" (UID: \"ca6f3e2f-0392-4d98-a47f-65deafaf93fb\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r" Oct 30 21:44:51.681426 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.681348 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d063b03-bc00-48bf-a91c-dfb31c286788-workload-certs\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4\" (UID: \"2d063b03-bc00-48bf-a91c-dfb31c286788\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4" Oct 30 21:44:51.681426 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.681383 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-workload-socket\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r\" (UID: \"ca6f3e2f-0392-4d98-a47f-65deafaf93fb\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r" Oct 30 21:44:51.681729 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.681464 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d063b03-bc00-48bf-a91c-dfb31c286788-istio-data\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4\" (UID: \"2d063b03-bc00-48bf-a91c-dfb31c286788\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4" Oct 30 21:44:51.681729 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.681506 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tflnp\" (UniqueName: \"kubernetes.io/projected/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-kube-api-access-tflnp\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r\" (UID: \"ca6f3e2f-0392-4d98-a47f-65deafaf93fb\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r" Oct 30 21:44:51.740127 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.739987 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4"] Oct 30 21:44:51.782643 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.782608 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d063b03-bc00-48bf-a91c-dfb31c286788-istiod-ca-cert\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4\" (UID: \"2d063b03-bc00-48bf-a91c-dfb31c286788\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4" Oct 30 21:44:51.782801 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.782657 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfw65\" (UniqueName: \"kubernetes.io/projected/2d063b03-bc00-48bf-a91c-dfb31c286788-kube-api-access-cfw65\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4\" (UID: \"2d063b03-bc00-48bf-a91c-dfb31c286788\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4" Oct 30 21:44:51.782801 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.782681 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-workload-certs\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r\" (UID: \"ca6f3e2f-0392-4d98-a47f-65deafaf93fb\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r" Oct 30 21:44:51.782801 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.782697 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-istio-podinfo\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r\" (UID: \"ca6f3e2f-0392-4d98-a47f-65deafaf93fb\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r" Oct 30 21:44:51.782801 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.782729 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d063b03-bc00-48bf-a91c-dfb31c286788-credential-socket\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4\" (UID: \"2d063b03-bc00-48bf-a91c-dfb31c286788\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4" Oct 30 21:44:51.782801 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.782746 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d063b03-bc00-48bf-a91c-dfb31c286788-istio-token\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4\" (UID: \"2d063b03-bc00-48bf-a91c-dfb31c286788\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4" Oct 30 21:44:51.782801 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.782796 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-istio-data\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r\" (UID: \"ca6f3e2f-0392-4d98-a47f-65deafaf93fb\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r" Oct 30 21:44:51.783211 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.782835 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d063b03-bc00-48bf-a91c-dfb31c286788-workload-socket\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4\" (UID: \"2d063b03-bc00-48bf-a91c-dfb31c286788\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4" Oct 30 21:44:51.783211 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.782870 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d063b03-bc00-48bf-a91c-dfb31c286788-istio-envoy\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4\" (UID: \"2d063b03-bc00-48bf-a91c-dfb31c286788\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4" Oct 30 21:44:51.783211 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.782905 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-credential-socket\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r\" (UID: \"ca6f3e2f-0392-4d98-a47f-65deafaf93fb\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r" Oct 30 21:44:51.783211 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.782933 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d063b03-bc00-48bf-a91c-dfb31c286788-workload-certs\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4\" (UID: \"2d063b03-bc00-48bf-a91c-dfb31c286788\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4" Oct 30 21:44:51.783211 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.782967 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-workload-socket\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r\" (UID: \"ca6f3e2f-0392-4d98-a47f-65deafaf93fb\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r" Oct 30 21:44:51.783211 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.783026 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d063b03-bc00-48bf-a91c-dfb31c286788-istio-data\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4\" (UID: \"2d063b03-bc00-48bf-a91c-dfb31c286788\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4" Oct 30 21:44:51.783211 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.783060 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tflnp\" (UniqueName: \"kubernetes.io/projected/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-kube-api-access-tflnp\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r\" (UID: \"ca6f3e2f-0392-4d98-a47f-65deafaf93fb\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r" Oct 30 21:44:51.783211 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.783145 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d063b03-bc00-48bf-a91c-dfb31c286788-istio-podinfo\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4\" (UID: \"2d063b03-bc00-48bf-a91c-dfb31c286788\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4" Oct 30 21:44:51.783211 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.783150 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-workload-certs\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r\" (UID: \"ca6f3e2f-0392-4d98-a47f-65deafaf93fb\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r" Oct 30 21:44:51.783211 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.783171 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-istiod-ca-cert\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r\" (UID: \"ca6f3e2f-0392-4d98-a47f-65deafaf93fb\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r" Oct 30 21:44:51.783753 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.783534 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d063b03-bc00-48bf-a91c-dfb31c286788-workload-certs\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4\" (UID: \"2d063b03-bc00-48bf-a91c-dfb31c286788\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4" Oct 30 21:44:51.783753 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.783626 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-istio-envoy\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r\" (UID: \"ca6f3e2f-0392-4d98-a47f-65deafaf93fb\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r" Oct 30 21:44:51.783753 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.783660 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-istio-token\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r\" (UID: \"ca6f3e2f-0392-4d98-a47f-65deafaf93fb\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r" Oct 30 21:44:51.783753 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.783678 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-workload-socket\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r\" (UID: \"ca6f3e2f-0392-4d98-a47f-65deafaf93fb\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r" Oct 30 21:44:51.784039 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.783971 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d063b03-bc00-48bf-a91c-dfb31c286788-istio-data\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4\" (UID: \"2d063b03-bc00-48bf-a91c-dfb31c286788\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4" Oct 30 21:44:51.784189 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.784169 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d063b03-bc00-48bf-a91c-dfb31c286788-workload-socket\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4\" (UID: \"2d063b03-bc00-48bf-a91c-dfb31c286788\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4" Oct 30 21:44:51.784280 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.784206 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d063b03-bc00-48bf-a91c-dfb31c286788-credential-socket\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4\" (UID: \"2d063b03-bc00-48bf-a91c-dfb31c286788\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4" Oct 30 21:44:51.784500 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.784481 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-istio-data\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r\" (UID: \"ca6f3e2f-0392-4d98-a47f-65deafaf93fb\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r" Oct 30 21:44:51.784580 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.784533 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-credential-socket\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r\" (UID: \"ca6f3e2f-0392-4d98-a47f-65deafaf93fb\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r" Oct 30 21:44:51.784689 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.784637 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-istiod-ca-cert\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r\" (UID: \"ca6f3e2f-0392-4d98-a47f-65deafaf93fb\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r" Oct 30 21:44:51.785081 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.785050 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d063b03-bc00-48bf-a91c-dfb31c286788-istiod-ca-cert\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4\" (UID: \"2d063b03-bc00-48bf-a91c-dfb31c286788\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4" Oct 30 21:44:51.786139 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.786077 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-istio-podinfo\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r\" (UID: \"ca6f3e2f-0392-4d98-a47f-65deafaf93fb\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r" Oct 30 21:44:51.786235 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.786188 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d063b03-bc00-48bf-a91c-dfb31c286788-istio-envoy\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4\" (UID: \"2d063b03-bc00-48bf-a91c-dfb31c286788\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4" Oct 30 21:44:51.786427 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.786403 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-istio-envoy\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r\" (UID: \"ca6f3e2f-0392-4d98-a47f-65deafaf93fb\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r" Oct 30 21:44:51.786766 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.786746 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d063b03-bc00-48bf-a91c-dfb31c286788-istio-podinfo\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4\" (UID: \"2d063b03-bc00-48bf-a91c-dfb31c286788\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4" Oct 30 21:44:51.793064 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.793043 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bc1bfa1-s9d7h"] Oct 30 21:44:51.798863 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.798844 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bc1bfa1-s9d7h" Oct 30 21:44:51.817797 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.817773 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8babaae-hl6tw" Oct 30 21:44:51.845686 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.845646 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tflnp\" (UniqueName: \"kubernetes.io/projected/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-kube-api-access-tflnp\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r\" (UID: \"ca6f3e2f-0392-4d98-a47f-65deafaf93fb\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r" Oct 30 21:44:51.845839 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.845821 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-istio-token\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r\" (UID: \"ca6f3e2f-0392-4d98-a47f-65deafaf93fb\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r" Oct 30 21:44:51.845977 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.845950 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfw65\" (UniqueName: \"kubernetes.io/projected/2d063b03-bc00-48bf-a91c-dfb31c286788-kube-api-access-cfw65\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4\" (UID: \"2d063b03-bc00-48bf-a91c-dfb31c286788\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4" Oct 30 21:44:51.846025 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.845977 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d063b03-bc00-48bf-a91c-dfb31c286788-istio-token\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4\" (UID: \"2d063b03-bc00-48bf-a91c-dfb31c286788\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4" Oct 30 21:44:51.855806 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.855778 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bc1bfa1-s9d7h"] Oct 30 21:44:51.884584 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.884556 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14682f40-c39a-4234-9d38-947c5db6201b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6bc1bfa1-s9d7h\" (UID: \"14682f40-c39a-4234-9d38-947c5db6201b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bc1bfa1-s9d7h" Oct 30 21:44:51.884713 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.884694 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14682f40-c39a-4234-9d38-947c5db6201b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6bc1bfa1-s9d7h\" (UID: \"14682f40-c39a-4234-9d38-947c5db6201b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bc1bfa1-s9d7h" Oct 30 21:44:51.884790 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.884736 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clfw8\" (UniqueName: \"kubernetes.io/projected/14682f40-c39a-4234-9d38-947c5db6201b-kube-api-access-clfw8\") pod \"maas-api-revoke-sub-6bc1bfa1-s9d7h\" (UID: \"14682f40-c39a-4234-9d38-947c5db6201b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bc1bfa1-s9d7h" Oct 30 21:44:51.933475 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.933443 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b9f950-cq6mz" Oct 30 21:44:51.937418 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.937392 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r" Oct 30 21:44:51.971786 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.971756 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4" Oct 30 21:44:51.989259 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.989177 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14682f40-c39a-4234-9d38-947c5db6201b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6bc1bfa1-s9d7h\" (UID: \"14682f40-c39a-4234-9d38-947c5db6201b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bc1bfa1-s9d7h" Oct 30 21:44:51.989650 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.989612 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14682f40-c39a-4234-9d38-947c5db6201b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6bc1bfa1-s9d7h\" (UID: \"14682f40-c39a-4234-9d38-947c5db6201b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bc1bfa1-s9d7h" Oct 30 21:44:51.989772 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.989658 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clfw8\" (UniqueName: \"kubernetes.io/projected/14682f40-c39a-4234-9d38-947c5db6201b-kube-api-access-clfw8\") pod \"maas-api-revoke-sub-6bc1bfa1-s9d7h\" (UID: \"14682f40-c39a-4234-9d38-947c5db6201b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bc1bfa1-s9d7h" Oct 30 21:44:51.989890 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.989868 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14682f40-c39a-4234-9d38-947c5db6201b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6bc1bfa1-s9d7h\" (UID: \"14682f40-c39a-4234-9d38-947c5db6201b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bc1bfa1-s9d7h" Oct 30 21:44:51.994700 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:51.994649 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14682f40-c39a-4234-9d38-947c5db6201b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6bc1bfa1-s9d7h\" (UID: \"14682f40-c39a-4234-9d38-947c5db6201b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bc1bfa1-s9d7h" Oct 30 21:44:52.051976 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:52.051913 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clfw8\" (UniqueName: \"kubernetes.io/projected/14682f40-c39a-4234-9d38-947c5db6201b-kube-api-access-clfw8\") pod \"maas-api-revoke-sub-6bc1bfa1-s9d7h\" (UID: \"14682f40-c39a-4234-9d38-947c5db6201b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bc1bfa1-s9d7h" Oct 30 21:44:52.117973 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:52.117931 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bc1bfa1-s9d7h" Oct 30 21:44:52.212964 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:44:52.212927 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod889613b9_6b8d_4ebc_a9ac_d9b2e9d8d817.slice/crio-e6c3b99ea8b5a51b6dccc1170f825816f21cf769e7fd27fa9280f1780d7ad36a WatchSource:0}: Error finding container e6c3b99ea8b5a51b6dccc1170f825816f21cf769e7fd27fa9280f1780d7ad36a: Status 404 returned error can't find the container with id e6c3b99ea8b5a51b6dccc1170f825816f21cf769e7fd27fa9280f1780d7ad36a Oct 30 21:44:52.213101 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:52.213016 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8babaae-hl6tw"] Oct 30 21:44:52.327735 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:52.327704 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02b9f950-cq6mz"] Oct 30 21:44:52.328215 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:44:52.328177 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4b02ecf_7f32_4555_a5b0_2821f21ad666.slice/crio-b11ef83496fa16da13aa61162a4c964f46cb8bd635feb79ce5a64c013d798cfd WatchSource:0}: Error finding container b11ef83496fa16da13aa61162a4c964f46cb8bd635feb79ce5a64c013d798cfd: Status 404 returned error can't find the container with id b11ef83496fa16da13aa61162a4c964f46cb8bd635feb79ce5a64c013d798cfd Oct 30 21:44:52.382607 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:52.382576 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r"] Oct 30 21:44:52.384070 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:44:52.384040 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca6f3e2f_0392_4d98_a47f_65deafaf93fb.slice/crio-662677d7f87d1d11e09c7d9286629ba71026a5a44afc0d3fd4731b84bdd5b644 WatchSource:0}: Error finding container 662677d7f87d1d11e09c7d9286629ba71026a5a44afc0d3fd4731b84bdd5b644: Status 404 returned error can't find the container with id 662677d7f87d1d11e09c7d9286629ba71026a5a44afc0d3fd4731b84bdd5b644 Oct 30 21:44:52.386480 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:52.386447 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:44:52.386583 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:52.386523 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:44:52.386583 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:52.386559 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:44:52.387714 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:52.387693 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4"] Oct 30 21:44:52.387997 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:44:52.387977 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d063b03_bc00_48bf_a91c_dfb31c286788.slice/crio-d7cabf8a695a2b17f2629a620802f08a729b21ee9303e6a135e690885a5aebd8 WatchSource:0}: Error finding container d7cabf8a695a2b17f2629a620802f08a729b21ee9303e6a135e690885a5aebd8: Status 404 returned error can't find the container with id d7cabf8a695a2b17f2629a620802f08a729b21ee9303e6a135e690885a5aebd8 Oct 30 21:44:52.390416 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:52.390387 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:44:52.390523 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:52.390463 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:44:52.390523 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:52.390507 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:44:52.502494 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:52.502456 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bc1bfa1-s9d7h"] Oct 30 21:44:52.513023 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:44:52.512945 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14682f40_c39a_4234_9d38_947c5db6201b.slice/crio-44ec8137462ce7a12c2fd764129f95b05b55ab6ce3dabe8bf547d1da3820ce3f WatchSource:0}: Error finding container 44ec8137462ce7a12c2fd764129f95b05b55ab6ce3dabe8bf547d1da3820ce3f: Status 404 returned error can't find the container with id 44ec8137462ce7a12c2fd764129f95b05b55ab6ce3dabe8bf547d1da3820ce3f Oct 30 21:44:52.663678 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:52.663638 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8babaae-hl6tw" event={"ID":"889613b9-6b8d-4ebc-a9ac-d9b2e9d8d817","Type":"ContainerStarted","Data":"e6c3b99ea8b5a51b6dccc1170f825816f21cf769e7fd27fa9280f1780d7ad36a"} Oct 30 21:44:52.665677 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:52.665644 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r" event={"ID":"ca6f3e2f-0392-4d98-a47f-65deafaf93fb","Type":"ContainerStarted","Data":"45bc329e0c1c9088945e82cf7033fec15f9e933b72d3825d2201bc6805353a5b"} Oct 30 21:44:52.665795 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:52.665696 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r" event={"ID":"ca6f3e2f-0392-4d98-a47f-65deafaf93fb","Type":"ContainerStarted","Data":"662677d7f87d1d11e09c7d9286629ba71026a5a44afc0d3fd4731b84bdd5b644"} Oct 30 21:44:52.666941 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:52.666917 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bc1bfa1-s9d7h" event={"ID":"14682f40-c39a-4234-9d38-947c5db6201b","Type":"ContainerStarted","Data":"44ec8137462ce7a12c2fd764129f95b05b55ab6ce3dabe8bf547d1da3820ce3f"} Oct 30 21:44:52.668371 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:52.668345 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b9f950-cq6mz" event={"ID":"d4b02ecf-7f32-4555-a5b0-2821f21ad666","Type":"ContainerStarted","Data":"b11ef83496fa16da13aa61162a4c964f46cb8bd635feb79ce5a64c013d798cfd"} Oct 30 21:44:52.670163 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:52.670136 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4" event={"ID":"2d063b03-bc00-48bf-a91c-dfb31c286788","Type":"ContainerStarted","Data":"3fb2a39df3ce04727343738955ed32b250dd0c4613daa99fee335f2bf00ccc25"} Oct 30 21:44:52.670297 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:52.670185 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4" event={"ID":"2d063b03-bc00-48bf-a91c-dfb31c286788","Type":"ContainerStarted","Data":"d7cabf8a695a2b17f2629a620802f08a729b21ee9303e6a135e690885a5aebd8"} Oct 30 21:44:52.791465 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:52.791346 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r" podStartSLOduration=1.791325474 podStartE2EDuration="1.791325474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:52.786810393 +0000 UTC m=+1276.529331329" watchObservedRunningTime="2026-10-30 21:44:52.791325474 +0000 UTC m=+1276.533846411" Oct 30 21:44:52.909042 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:52.908984 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4" podStartSLOduration=1.90896603 podStartE2EDuration="1.90896603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:52.908197055 +0000 UTC m=+1276.650717991" watchObservedRunningTime="2026-10-30 21:44:52.90896603 +0000 UTC m=+1276.651486974" Oct 30 21:44:52.939920 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:52.939001 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r" Oct 30 21:44:52.975498 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:52.975453 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4" Oct 30 21:44:53.679516 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:53.679486 5748 generic.go:356] "Generic (PLEG): container finished" podID="889613b9-6b8d-4ebc-a9ac-d9b2e9d8d817" containerID="a5df42c28aa41493c6daf404e0bfc49a7deea5bc943e402899aa4176f7f77cd2" exitCode=0 Oct 30 21:44:53.680052 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:53.679673 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8babaae-hl6tw" event={"ID":"889613b9-6b8d-4ebc-a9ac-d9b2e9d8d817","Type":"ContainerDied","Data":"a5df42c28aa41493c6daf404e0bfc49a7deea5bc943e402899aa4176f7f77cd2"} Oct 30 21:44:53.681586 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:53.681563 5748 generic.go:356] "Generic (PLEG): container finished" podID="14682f40-c39a-4234-9d38-947c5db6201b" containerID="cabc765874f70aad7084fba3b8930abd9162d35fb4cf593efca21bca0f83817f" exitCode=0 Oct 30 21:44:53.681688 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:53.681673 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bc1bfa1-s9d7h" event={"ID":"14682f40-c39a-4234-9d38-947c5db6201b","Type":"ContainerDied","Data":"cabc765874f70aad7084fba3b8930abd9162d35fb4cf593efca21bca0f83817f"} Oct 30 21:44:53.683202 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:53.683180 5748 generic.go:356] "Generic (PLEG): container finished" podID="d4b02ecf-7f32-4555-a5b0-2821f21ad666" containerID="6a13cbee880b07da5903a07708f9af93c9748134e4e509743a46363e6c164313" exitCode=0 Oct 30 21:44:53.683957 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:53.683940 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b9f950-cq6mz" event={"ID":"d4b02ecf-7f32-4555-a5b0-2821f21ad666","Type":"ContainerDied","Data":"6a13cbee880b07da5903a07708f9af93c9748134e4e509743a46363e6c164313"} Oct 30 21:44:53.953453 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:53.953339 5748 patch_prober.go:29] interesting pod/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 21:44:53.953718 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:53.953425 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r" podUID="ca6f3e2f-0392-4d98-a47f-65deafaf93fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 21:44:53.977175 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:53.977142 5748 patch_prober.go:29] interesting pod/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 21:44:53.977345 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:53.977193 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4" podUID="2d063b03-bc00-48bf-a91c-dfb31c286788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 21:44:54.954122 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:54.940320 5748 patch_prober.go:29] interesting pod/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 21:44:54.954122 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:54.940387 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r" podUID="ca6f3e2f-0392-4d98-a47f-65deafaf93fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 21:44:54.990485 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:54.987033 5748 patch_prober.go:29] interesting pod/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 21:44:54.990485 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:54.987092 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4" podUID="2d063b03-bc00-48bf-a91c-dfb31c286788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 21:44:55.070433 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:55.069980 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5fe7552-mdv9r"] Oct 30 21:44:55.074924 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:55.074259 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5fe7552-mdv9r"] Oct 30 21:44:55.167362 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:55.167337 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8babaae-hl6tw" Oct 30 21:44:55.196683 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:55.196651 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/889613b9-6b8d-4ebc-a9ac-d9b2e9d8d817-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/889613b9-6b8d-4ebc-a9ac-d9b2e9d8d817-maas-api-xks-ca\") pod \"889613b9-6b8d-4ebc-a9ac-d9b2e9d8d817\" (UID: \"889613b9-6b8d-4ebc-a9ac-d9b2e9d8d817\") " Oct 30 21:44:55.197118 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:55.197093 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/889613b9-6b8d-4ebc-a9ac-d9b2e9d8d817-kube-api-access-zrgpb\" (UniqueName: \"kubernetes.io/projected/889613b9-6b8d-4ebc-a9ac-d9b2e9d8d817-kube-api-access-zrgpb\") pod \"889613b9-6b8d-4ebc-a9ac-d9b2e9d8d817\" (UID: \"889613b9-6b8d-4ebc-a9ac-d9b2e9d8d817\") " Oct 30 21:44:55.197314 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:55.197304 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/889613b9-6b8d-4ebc-a9ac-d9b2e9d8d817-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/889613b9-6b8d-4ebc-a9ac-d9b2e9d8d817-maas-api-openshift-ca\") pod \"889613b9-6b8d-4ebc-a9ac-d9b2e9d8d817\" (UID: \"889613b9-6b8d-4ebc-a9ac-d9b2e9d8d817\") " Oct 30 21:44:55.198492 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:55.198392 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/889613b9-6b8d-4ebc-a9ac-d9b2e9d8d817-maas-api-openshift-ca" pod "889613b9-6b8d-4ebc-a9ac-d9b2e9d8d817" (UID: "889613b9-6b8d-4ebc-a9ac-d9b2e9d8d817"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:44:55.199323 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:55.199296 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/889613b9-6b8d-4ebc-a9ac-d9b2e9d8d817-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:55.208668 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:55.208418 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/889613b9-6b8d-4ebc-a9ac-d9b2e9d8d817-kube-api-access-zrgpb" pod "889613b9-6b8d-4ebc-a9ac-d9b2e9d8d817" (UID: "889613b9-6b8d-4ebc-a9ac-d9b2e9d8d817"). InnerVolumeSpecName "kube-api-access-zrgpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:55.268941 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:55.268079 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/889613b9-6b8d-4ebc-a9ac-d9b2e9d8d817-maas-api-xks-ca" pod "889613b9-6b8d-4ebc-a9ac-d9b2e9d8d817" (UID: "889613b9-6b8d-4ebc-a9ac-d9b2e9d8d817"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:44:55.274609 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:55.274582 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b9f950-cq6mz" Oct 30 21:44:55.277982 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:55.277962 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bc1bfa1-s9d7h" Oct 30 21:44:55.303922 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:55.301166 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14682f40-c39a-4234-9d38-947c5db6201b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14682f40-c39a-4234-9d38-947c5db6201b-maas-api-xks-ca\") pod \"14682f40-c39a-4234-9d38-947c5db6201b\" (UID: \"14682f40-c39a-4234-9d38-947c5db6201b\") " Oct 30 21:44:55.303922 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:55.301298 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4b02ecf-7f32-4555-a5b0-2821f21ad666-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4b02ecf-7f32-4555-a5b0-2821f21ad666-maas-api-xks-ca\") pod \"d4b02ecf-7f32-4555-a5b0-2821f21ad666\" (UID: \"d4b02ecf-7f32-4555-a5b0-2821f21ad666\") " Oct 30 21:44:55.303922 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:55.301333 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14682f40-c39a-4234-9d38-947c5db6201b-kube-api-access-clfw8\" (UniqueName: \"kubernetes.io/projected/14682f40-c39a-4234-9d38-947c5db6201b-kube-api-access-clfw8\") pod \"14682f40-c39a-4234-9d38-947c5db6201b\" (UID: \"14682f40-c39a-4234-9d38-947c5db6201b\") " Oct 30 21:44:55.303922 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:55.301399 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4b02ecf-7f32-4555-a5b0-2821f21ad666-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4b02ecf-7f32-4555-a5b0-2821f21ad666-maas-api-openshift-ca\") pod \"d4b02ecf-7f32-4555-a5b0-2821f21ad666\" (UID: \"d4b02ecf-7f32-4555-a5b0-2821f21ad666\") " Oct 30 21:44:55.303922 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:55.301486 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4b02ecf-7f32-4555-a5b0-2821f21ad666-kube-api-access-lfksk\" (UniqueName: \"kubernetes.io/projected/d4b02ecf-7f32-4555-a5b0-2821f21ad666-kube-api-access-lfksk\") pod \"d4b02ecf-7f32-4555-a5b0-2821f21ad666\" (UID: \"d4b02ecf-7f32-4555-a5b0-2821f21ad666\") " Oct 30 21:44:55.303922 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:55.301515 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14682f40-c39a-4234-9d38-947c5db6201b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14682f40-c39a-4234-9d38-947c5db6201b-maas-api-openshift-ca\") pod \"14682f40-c39a-4234-9d38-947c5db6201b\" (UID: \"14682f40-c39a-4234-9d38-947c5db6201b\") " Oct 30 21:44:55.303922 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:55.302019 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/889613b9-6b8d-4ebc-a9ac-d9b2e9d8d817-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:55.303922 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:55.302043 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrgpb\" (UniqueName: \"kubernetes.io/projected/889613b9-6b8d-4ebc-a9ac-d9b2e9d8d817-kube-api-access-zrgpb\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:55.321929 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:55.315109 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14682f40-c39a-4234-9d38-947c5db6201b-maas-api-openshift-ca" pod "14682f40-c39a-4234-9d38-947c5db6201b" (UID: "14682f40-c39a-4234-9d38-947c5db6201b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:44:55.321929 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:55.320113 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4b02ecf-7f32-4555-a5b0-2821f21ad666-maas-api-openshift-ca" pod "d4b02ecf-7f32-4555-a5b0-2821f21ad666" (UID: "d4b02ecf-7f32-4555-a5b0-2821f21ad666"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:44:55.350922 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:55.350526 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14682f40-c39a-4234-9d38-947c5db6201b-kube-api-access-clfw8" pod "14682f40-c39a-4234-9d38-947c5db6201b" (UID: "14682f40-c39a-4234-9d38-947c5db6201b"). InnerVolumeSpecName "kube-api-access-clfw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:55.353512 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:55.351589 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14682f40-c39a-4234-9d38-947c5db6201b-maas-api-xks-ca" pod "14682f40-c39a-4234-9d38-947c5db6201b" (UID: "14682f40-c39a-4234-9d38-947c5db6201b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:44:55.353625 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:55.353582 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4b02ecf-7f32-4555-a5b0-2821f21ad666-maas-api-xks-ca" pod "d4b02ecf-7f32-4555-a5b0-2821f21ad666" (UID: "d4b02ecf-7f32-4555-a5b0-2821f21ad666"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:44:55.354018 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:55.353990 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4b02ecf-7f32-4555-a5b0-2821f21ad666-kube-api-access-lfksk" pod "d4b02ecf-7f32-4555-a5b0-2821f21ad666" (UID: "d4b02ecf-7f32-4555-a5b0-2821f21ad666"). InnerVolumeSpecName "kube-api-access-lfksk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:55.436923 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:55.426060 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14682f40-c39a-4234-9d38-947c5db6201b-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:55.436923 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:55.426110 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4b02ecf-7f32-4555-a5b0-2821f21ad666-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:55.436923 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:55.426128 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clfw8\" (UniqueName: \"kubernetes.io/projected/14682f40-c39a-4234-9d38-947c5db6201b-kube-api-access-clfw8\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:55.436923 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:55.426142 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4b02ecf-7f32-4555-a5b0-2821f21ad666-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:55.436923 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:55.426156 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfksk\" (UniqueName: \"kubernetes.io/projected/d4b02ecf-7f32-4555-a5b0-2821f21ad666-kube-api-access-lfksk\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:55.436923 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:55.426169 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14682f40-c39a-4234-9d38-947c5db6201b-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:55.699630 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:55.699592 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bc1bfa1-s9d7h" Oct 30 21:44:55.699630 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:55.699612 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bc1bfa1-s9d7h" event={"ID":"14682f40-c39a-4234-9d38-947c5db6201b","Type":"ContainerDied","Data":"44ec8137462ce7a12c2fd764129f95b05b55ab6ce3dabe8bf547d1da3820ce3f"} Oct 30 21:44:55.699824 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:55.699665 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44ec8137462ce7a12c2fd764129f95b05b55ab6ce3dabe8bf547d1da3820ce3f" Oct 30 21:44:55.748922 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:55.748220 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b9f950-cq6mz" Oct 30 21:44:55.748922 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:55.748331 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b9f950-cq6mz" event={"ID":"d4b02ecf-7f32-4555-a5b0-2821f21ad666","Type":"ContainerDied","Data":"b11ef83496fa16da13aa61162a4c964f46cb8bd635feb79ce5a64c013d798cfd"} Oct 30 21:44:55.748922 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:55.748386 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b11ef83496fa16da13aa61162a4c964f46cb8bd635feb79ce5a64c013d798cfd" Oct 30 21:44:55.751820 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:55.751772 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8babaae-hl6tw" event={"ID":"889613b9-6b8d-4ebc-a9ac-d9b2e9d8d817","Type":"ContainerDied","Data":"e6c3b99ea8b5a51b6dccc1170f825816f21cf769e7fd27fa9280f1780d7ad36a"} Oct 30 21:44:55.751966 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:55.751844 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6c3b99ea8b5a51b6dccc1170f825816f21cf769e7fd27fa9280f1780d7ad36a" Oct 30 21:44:55.751966 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:55.751950 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8babaae-hl6tw" Oct 30 21:44:55.938325 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:55.938287 5748 patch_prober.go:29] interesting pod/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 21:44:55.938599 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:55.938570 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r" podUID="ca6f3e2f-0392-4d98-a47f-65deafaf93fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 21:44:55.975447 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:55.975367 5748 patch_prober.go:29] interesting pod/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 21:44:55.975447 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:55.975431 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4" podUID="2d063b03-bc00-48bf-a91c-dfb31c286788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 21:44:56.460927 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:56.458915 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="559e85df-748d-447a-9adc-3975200f7895" path="/var/lib/kubelet/pods/559e85df-748d-447a-9adc-3975200f7895/volumes" Oct 30 21:44:56.938921 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:56.938285 5748 patch_prober.go:29] interesting pod/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 21:44:56.938921 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:56.938347 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r" podUID="ca6f3e2f-0392-4d98-a47f-65deafaf93fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 21:44:56.977169 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:56.975469 5748 patch_prober.go:29] interesting pod/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 21:44:56.977169 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:56.975518 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4" podUID="2d063b03-bc00-48bf-a91c-dfb31c286788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 21:44:57.947921 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:57.944348 5748 patch_prober.go:29] interesting pod/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 21:44:57.947921 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:57.944416 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r" podUID="ca6f3e2f-0392-4d98-a47f-65deafaf93fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 21:44:57.975499 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:57.975464 5748 patch_prober.go:29] interesting pod/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 21:44:57.975638 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:57.975520 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4" podUID="2d063b03-bc00-48bf-a91c-dfb31c286788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 21:44:58.356347 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:58.356258 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d69ca2e6-dx6dj"] Oct 30 21:44:58.356952 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:58.356934 5748 cpu_manager.go:414] "containerMap: removing container" podUID="d4b02ecf-7f32-4555-a5b0-2821f21ad666" containerName="revoke-keys" Oct 30 21:44:58.357010 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:58.356953 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4b02ecf-7f32-4555-a5b0-2821f21ad666" containerName="revoke-keys" Oct 30 21:44:58.357010 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:58.356974 5748 cpu_manager.go:414] "containerMap: removing container" podUID="889613b9-6b8d-4ebc-a9ac-d9b2e9d8d817" containerName="revoke-keys" Oct 30 21:44:58.357010 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:58.356979 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="889613b9-6b8d-4ebc-a9ac-d9b2e9d8d817" containerName="revoke-keys" Oct 30 21:44:58.357010 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:58.356992 5748 cpu_manager.go:414] "containerMap: removing container" podUID="14682f40-c39a-4234-9d38-947c5db6201b" containerName="revoke-keys" Oct 30 21:44:58.357010 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:58.356997 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14682f40-c39a-4234-9d38-947c5db6201b" containerName="revoke-keys" Oct 30 21:44:58.357155 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:58.357119 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4b02ecf-7f32-4555-a5b0-2821f21ad666" containerName="revoke-keys" Oct 30 21:44:58.357155 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:58.357129 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="889613b9-6b8d-4ebc-a9ac-d9b2e9d8d817" containerName="revoke-keys" Oct 30 21:44:58.357155 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:58.357137 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="14682f40-c39a-4234-9d38-947c5db6201b" containerName="revoke-keys" Oct 30 21:44:58.361708 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:58.361689 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d69ca2e6-dx6dj" Oct 30 21:44:58.388215 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:58.388178 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7prq\" (UniqueName: \"kubernetes.io/projected/fe58c836-3dbd-4389-83c8-f66b58152106-kube-api-access-t7prq\") pod \"maas-api-revoke-sub-d69ca2e6-dx6dj\" (UID: \"fe58c836-3dbd-4389-83c8-f66b58152106\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d69ca2e6-dx6dj" Oct 30 21:44:58.388397 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:58.388374 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe58c836-3dbd-4389-83c8-f66b58152106-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d69ca2e6-dx6dj\" (UID: \"fe58c836-3dbd-4389-83c8-f66b58152106\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d69ca2e6-dx6dj" Oct 30 21:44:58.388436 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:58.388417 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe58c836-3dbd-4389-83c8-f66b58152106-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d69ca2e6-dx6dj\" (UID: \"fe58c836-3dbd-4389-83c8-f66b58152106\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d69ca2e6-dx6dj" Oct 30 21:44:58.460937 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:58.460891 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:44:58.462002 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:58.460989 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 21:44:58.462002 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:58.461034 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:44:58.462002 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:58.461101 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:44:58.466164 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:58.466125 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d69ca2e6-dx6dj"] Oct 30 21:44:58.489399 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:58.489366 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe58c836-3dbd-4389-83c8-f66b58152106-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d69ca2e6-dx6dj\" (UID: \"fe58c836-3dbd-4389-83c8-f66b58152106\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d69ca2e6-dx6dj" Oct 30 21:44:58.489510 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:58.489428 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe58c836-3dbd-4389-83c8-f66b58152106-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d69ca2e6-dx6dj\" (UID: \"fe58c836-3dbd-4389-83c8-f66b58152106\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d69ca2e6-dx6dj" Oct 30 21:44:58.489573 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:58.489554 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7prq\" (UniqueName: \"kubernetes.io/projected/fe58c836-3dbd-4389-83c8-f66b58152106-kube-api-access-t7prq\") pod \"maas-api-revoke-sub-d69ca2e6-dx6dj\" (UID: \"fe58c836-3dbd-4389-83c8-f66b58152106\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d69ca2e6-dx6dj" Oct 30 21:44:58.490408 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:58.490384 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe58c836-3dbd-4389-83c8-f66b58152106-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d69ca2e6-dx6dj\" (UID: \"fe58c836-3dbd-4389-83c8-f66b58152106\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d69ca2e6-dx6dj" Oct 30 21:44:58.492792 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:58.492772 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe58c836-3dbd-4389-83c8-f66b58152106-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d69ca2e6-dx6dj\" (UID: \"fe58c836-3dbd-4389-83c8-f66b58152106\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d69ca2e6-dx6dj" Oct 30 21:44:58.552302 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:58.552259 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7prq\" (UniqueName: \"kubernetes.io/projected/fe58c836-3dbd-4389-83c8-f66b58152106-kube-api-access-t7prq\") pod \"maas-api-revoke-sub-d69ca2e6-dx6dj\" (UID: \"fe58c836-3dbd-4389-83c8-f66b58152106\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d69ca2e6-dx6dj" Oct 30 21:44:58.675728 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:58.675687 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d69ca2e6-dx6dj" Oct 30 21:44:58.938300 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:58.938214 5748 patch_prober.go:29] interesting pod/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 21:44:58.938300 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:58.938281 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r" podUID="ca6f3e2f-0392-4d98-a47f-65deafaf93fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 21:44:58.975775 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:58.975739 5748 patch_prober.go:29] interesting pod/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 21:44:58.975975 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:58.975795 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4" podUID="2d063b03-bc00-48bf-a91c-dfb31c286788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 21:44:59.069622 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:59.069569 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d69ca2e6-dx6dj"] Oct 30 21:44:59.073455 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:44:59.073413 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe58c836_3dbd_4389_83c8_f66b58152106.slice/crio-17150878b0cbab22fb763bfa1b896f3e0416780fd7776dcc2fc073b099fc0a4e WatchSource:0}: Error finding container 17150878b0cbab22fb763bfa1b896f3e0416780fd7776dcc2fc073b099fc0a4e: Status 404 returned error can't find the container with id 17150878b0cbab22fb763bfa1b896f3e0416780fd7776dcc2fc073b099fc0a4e Oct 30 21:44:59.794433 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:59.794388 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d69ca2e6-dx6dj" event={"ID":"fe58c836-3dbd-4389-83c8-f66b58152106","Type":"ContainerStarted","Data":"7c2944c7d238e05045aa754e0ad50e853a0e09555d89c599cdf05d2f8779ff7f"} Oct 30 21:44:59.794996 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:59.794466 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d69ca2e6-dx6dj" event={"ID":"fe58c836-3dbd-4389-83c8-f66b58152106","Type":"ContainerStarted","Data":"17150878b0cbab22fb763bfa1b896f3e0416780fd7776dcc2fc073b099fc0a4e"} Oct 30 21:44:59.914135 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:59.914079 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d69ca2e6-dx6dj" podStartSLOduration=1.914066026 podStartE2EDuration="1.914066026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:59.91244557 +0000 UTC m=+1283.654966516" watchObservedRunningTime="2026-10-30 21:44:59.914066026 +0000 UTC m=+1283.656587008" Oct 30 21:44:59.942703 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:59.942670 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r" Oct 30 21:44:59.943105 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:59.943084 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r" Oct 30 21:44:59.943858 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:59.943842 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r" Oct 30 21:44:59.979822 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:59.979794 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4" Oct 30 21:44:59.980103 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:59.980077 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4" Oct 30 21:44:59.980927 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:44:59.980887 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4" Oct 30 21:45:00.049048 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:00.048967 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7277fd6-62shf"] Oct 30 21:45:00.052599 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:00.052569 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7277fd6-62shf"] Oct 30 21:45:00.147952 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:45:00.147915 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:45:00.163051 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:00.163018 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889945-6lzds"] Oct 30 21:45:00.167551 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:00.167529 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889945-6lzds" Oct 30 21:45:00.167676 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:00.167564 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91874e-29889945-hbl8j"] Oct 30 21:45:00.172401 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:00.172379 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d68156-29889945-2nc2b"] Oct 30 21:45:00.172516 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:00.172484 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91874e-29889945-hbl8j" Oct 30 21:45:00.176686 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:00.176668 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d68156-29889945-2nc2b" Oct 30 21:45:00.185210 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:00.184856 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-691598-29889945-vzhl4"] Oct 30 21:45:00.190874 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:00.190842 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-691598-29889945-vzhl4" Oct 30 21:45:00.217533 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:00.217499 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28a0c85c-d54f-42f9-8c57-1508fa1b2c82-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889945-6lzds\" (UID: \"28a0c85c-d54f-42f9-8c57-1508fa1b2c82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889945-6lzds" Oct 30 21:45:00.217683 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:00.217561 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30fdce00-7aeb-4812-a580-8dde48515851-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-d68156-29889945-2nc2b\" (UID: \"30fdce00-7aeb-4812-a580-8dde48515851\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d68156-29889945-2nc2b" Oct 30 21:45:00.217683 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:00.217628 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-596pn\" (UniqueName: \"kubernetes.io/projected/30fdce00-7aeb-4812-a580-8dde48515851-kube-api-access-596pn\") pod \"maas-api-key-cleanup-e2e-worker-w3-d68156-29889945-2nc2b\" (UID: \"30fdce00-7aeb-4812-a580-8dde48515851\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d68156-29889945-2nc2b" Oct 30 21:45:00.217683 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:00.217662 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dba2ae0f-a9ff-48da-b2e6-f8011473d615-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-91874e-29889945-hbl8j\" (UID: \"dba2ae0f-a9ff-48da-b2e6-f8011473d615\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91874e-29889945-hbl8j" Oct 30 21:45:00.217861 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:00.217699 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpkfz\" (UniqueName: \"kubernetes.io/projected/dba2ae0f-a9ff-48da-b2e6-f8011473d615-kube-api-access-bpkfz\") pod \"maas-api-key-cleanup-e2e-worker-w2-91874e-29889945-hbl8j\" (UID: \"dba2ae0f-a9ff-48da-b2e6-f8011473d615\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91874e-29889945-hbl8j" Oct 30 21:45:00.217861 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:00.217774 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6gnp\" (UniqueName: \"kubernetes.io/projected/28a0c85c-d54f-42f9-8c57-1508fa1b2c82-kube-api-access-f6gnp\") pod \"maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889945-6lzds\" (UID: \"28a0c85c-d54f-42f9-8c57-1508fa1b2c82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889945-6lzds" Oct 30 21:45:00.233178 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:00.231881 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91874e-29889945-hbl8j"] Oct 30 21:45:00.233178 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:00.232046 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889945-6lzds"] Oct 30 21:45:00.270117 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:00.270090 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d68156-29889945-2nc2b"] Oct 30 21:45:00.275524 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:00.275494 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-691598-29889945-vzhl4"] Oct 30 21:45:00.324962 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:00.322280 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-596pn\" (UniqueName: \"kubernetes.io/projected/30fdce00-7aeb-4812-a580-8dde48515851-kube-api-access-596pn\") pod \"maas-api-key-cleanup-e2e-worker-w3-d68156-29889945-2nc2b\" (UID: \"30fdce00-7aeb-4812-a580-8dde48515851\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d68156-29889945-2nc2b" Oct 30 21:45:00.324962 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:00.322377 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dba2ae0f-a9ff-48da-b2e6-f8011473d615-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-91874e-29889945-hbl8j\" (UID: \"dba2ae0f-a9ff-48da-b2e6-f8011473d615\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91874e-29889945-hbl8j" Oct 30 21:45:00.324962 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:00.322428 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpkfz\" (UniqueName: \"kubernetes.io/projected/dba2ae0f-a9ff-48da-b2e6-f8011473d615-kube-api-access-bpkfz\") pod \"maas-api-key-cleanup-e2e-worker-w2-91874e-29889945-hbl8j\" (UID: \"dba2ae0f-a9ff-48da-b2e6-f8011473d615\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91874e-29889945-hbl8j" Oct 30 21:45:00.324962 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:00.322548 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6gnp\" (UniqueName: \"kubernetes.io/projected/28a0c85c-d54f-42f9-8c57-1508fa1b2c82-kube-api-access-f6gnp\") pod \"maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889945-6lzds\" (UID: \"28a0c85c-d54f-42f9-8c57-1508fa1b2c82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889945-6lzds" Oct 30 21:45:00.324962 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:00.322720 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c042e96b-236e-419e-b531-43798f9bb25f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-691598-29889945-vzhl4\" (UID: \"c042e96b-236e-419e-b531-43798f9bb25f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-691598-29889945-vzhl4" Oct 30 21:45:00.324962 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:00.323200 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28a0c85c-d54f-42f9-8c57-1508fa1b2c82-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889945-6lzds\" (UID: \"28a0c85c-d54f-42f9-8c57-1508fa1b2c82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889945-6lzds" Oct 30 21:45:00.324962 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:00.323351 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30fdce00-7aeb-4812-a580-8dde48515851-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-d68156-29889945-2nc2b\" (UID: \"30fdce00-7aeb-4812-a580-8dde48515851\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d68156-29889945-2nc2b" Oct 30 21:45:00.324962 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:00.323488 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx9w8\" (UniqueName: \"kubernetes.io/projected/c042e96b-236e-419e-b531-43798f9bb25f-kube-api-access-sx9w8\") pod \"maas-api-key-cleanup-e2e-worker-w0-691598-29889945-vzhl4\" (UID: \"c042e96b-236e-419e-b531-43798f9bb25f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-691598-29889945-vzhl4" Oct 30 21:45:00.330912 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:00.326507 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dba2ae0f-a9ff-48da-b2e6-f8011473d615-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-91874e-29889945-hbl8j\" (UID: \"dba2ae0f-a9ff-48da-b2e6-f8011473d615\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91874e-29889945-hbl8j" Oct 30 21:45:00.330912 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:00.329006 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28a0c85c-d54f-42f9-8c57-1508fa1b2c82-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889945-6lzds\" (UID: \"28a0c85c-d54f-42f9-8c57-1508fa1b2c82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889945-6lzds" Oct 30 21:45:00.330912 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:00.329577 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30fdce00-7aeb-4812-a580-8dde48515851-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-d68156-29889945-2nc2b\" (UID: \"30fdce00-7aeb-4812-a580-8dde48515851\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d68156-29889945-2nc2b" Oct 30 21:45:00.386767 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:00.386730 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-596pn\" (UniqueName: \"kubernetes.io/projected/30fdce00-7aeb-4812-a580-8dde48515851-kube-api-access-596pn\") pod \"maas-api-key-cleanup-e2e-worker-w3-d68156-29889945-2nc2b\" (UID: \"30fdce00-7aeb-4812-a580-8dde48515851\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d68156-29889945-2nc2b" Oct 30 21:45:00.390280 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:00.390253 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpkfz\" (UniqueName: \"kubernetes.io/projected/dba2ae0f-a9ff-48da-b2e6-f8011473d615-kube-api-access-bpkfz\") pod \"maas-api-key-cleanup-e2e-worker-w2-91874e-29889945-hbl8j\" (UID: \"dba2ae0f-a9ff-48da-b2e6-f8011473d615\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91874e-29889945-hbl8j" Oct 30 21:45:00.393153 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:00.393131 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6gnp\" (UniqueName: \"kubernetes.io/projected/28a0c85c-d54f-42f9-8c57-1508fa1b2c82-kube-api-access-f6gnp\") pod \"maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889945-6lzds\" (UID: \"28a0c85c-d54f-42f9-8c57-1508fa1b2c82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889945-6lzds" Oct 30 21:45:00.420873 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:00.420839 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ea379a7-6815-4d5a-82ce-f87e0ae086c3" path="/var/lib/kubelet/pods/1ea379a7-6815-4d5a-82ce-f87e0ae086c3/volumes" Oct 30 21:45:00.424831 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:00.424804 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c042e96b-236e-419e-b531-43798f9bb25f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-691598-29889945-vzhl4\" (UID: \"c042e96b-236e-419e-b531-43798f9bb25f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-691598-29889945-vzhl4" Oct 30 21:45:00.424977 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:00.424945 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx9w8\" (UniqueName: \"kubernetes.io/projected/c042e96b-236e-419e-b531-43798f9bb25f-kube-api-access-sx9w8\") pod \"maas-api-key-cleanup-e2e-worker-w0-691598-29889945-vzhl4\" (UID: \"c042e96b-236e-419e-b531-43798f9bb25f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-691598-29889945-vzhl4" Oct 30 21:45:00.425781 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:00.425756 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c042e96b-236e-419e-b531-43798f9bb25f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-691598-29889945-vzhl4\" (UID: \"c042e96b-236e-419e-b531-43798f9bb25f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-691598-29889945-vzhl4" Oct 30 21:45:00.486386 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:00.486346 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx9w8\" (UniqueName: \"kubernetes.io/projected/c042e96b-236e-419e-b531-43798f9bb25f-kube-api-access-sx9w8\") pod \"maas-api-key-cleanup-e2e-worker-w0-691598-29889945-vzhl4\" (UID: \"c042e96b-236e-419e-b531-43798f9bb25f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-691598-29889945-vzhl4" Oct 30 21:45:00.502256 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:00.501165 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889945-6lzds" Oct 30 21:45:00.549649 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:00.549616 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91874e-29889945-hbl8j" Oct 30 21:45:00.575670 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:00.573929 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d68156-29889945-2nc2b" Oct 30 21:45:00.583268 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:00.583243 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-691598-29889945-vzhl4" Oct 30 21:45:00.806062 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:00.806031 5748 generic.go:356] "Generic (PLEG): container finished" podID="fe58c836-3dbd-4389-83c8-f66b58152106" containerID="7c2944c7d238e05045aa754e0ad50e853a0e09555d89c599cdf05d2f8779ff7f" exitCode=0 Oct 30 21:45:00.806490 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:00.806116 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d69ca2e6-dx6dj" event={"ID":"fe58c836-3dbd-4389-83c8-f66b58152106","Type":"ContainerDied","Data":"7c2944c7d238e05045aa754e0ad50e853a0e09555d89c599cdf05d2f8779ff7f"} Oct 30 21:45:00.899583 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:00.899553 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889945-6lzds"] Oct 30 21:45:00.899997 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:45:00.899963 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28a0c85c_d54f_42f9_8c57_1508fa1b2c82.slice/crio-e52d9c1d65ab42163336a142295af1a0ac6a72c36ba4016bf753a79f63b74881 WatchSource:0}: Error finding container e52d9c1d65ab42163336a142295af1a0ac6a72c36ba4016bf753a79f63b74881: Status 404 returned error can't find the container with id e52d9c1d65ab42163336a142295af1a0ac6a72c36ba4016bf753a79f63b74881 Oct 30 21:45:00.966828 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:45:00.966797 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddba2ae0f_a9ff_48da_b2e6_f8011473d615.slice/crio-4b47561863418238acca62312e1a17d4942c1beab182479336b1ba90e0477fb9 WatchSource:0}: Error finding container 4b47561863418238acca62312e1a17d4942c1beab182479336b1ba90e0477fb9: Status 404 returned error can't find the container with id 4b47561863418238acca62312e1a17d4942c1beab182479336b1ba90e0477fb9 Oct 30 21:45:00.967414 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:00.967351 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91874e-29889945-hbl8j"] Oct 30 21:45:01.016633 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:01.016603 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d68156-29889945-2nc2b"] Oct 30 21:45:01.017678 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:45:01.017653 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30fdce00_7aeb_4812_a580_8dde48515851.slice/crio-6b18a371cc0579afa521d73823d30cfc272f6719d4dc5b1f5a103b0d37e40643 WatchSource:0}: Error finding container 6b18a371cc0579afa521d73823d30cfc272f6719d4dc5b1f5a103b0d37e40643: Status 404 returned error can't find the container with id 6b18a371cc0579afa521d73823d30cfc272f6719d4dc5b1f5a103b0d37e40643 Oct 30 21:45:01.031602 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:01.031569 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-691598-29889945-vzhl4"] Oct 30 21:45:01.033542 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:45:01.033511 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc042e96b_236e_419e_b531_43798f9bb25f.slice/crio-de51a9b27b4ce608b8c7e17908edceccc01c0fae60f37b36450387adb06364ff WatchSource:0}: Error finding container de51a9b27b4ce608b8c7e17908edceccc01c0fae60f37b36450387adb06364ff: Status 404 returned error can't find the container with id de51a9b27b4ce608b8c7e17908edceccc01c0fae60f37b36450387adb06364ff Oct 30 21:45:01.817859 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:01.817827 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-691598-29889945-vzhl4" event={"ID":"c042e96b-236e-419e-b531-43798f9bb25f","Type":"ContainerStarted","Data":"71d37b7c49c78323e7d722a811ec6105096d9e4ac24adb0bada22bc8e0bbd8e3"} Oct 30 21:45:01.818307 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:01.817886 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-691598-29889945-vzhl4" event={"ID":"c042e96b-236e-419e-b531-43798f9bb25f","Type":"ContainerStarted","Data":"de51a9b27b4ce608b8c7e17908edceccc01c0fae60f37b36450387adb06364ff"} Oct 30 21:45:01.819471 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:01.819447 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d68156-29889945-2nc2b" event={"ID":"30fdce00-7aeb-4812-a580-8dde48515851","Type":"ContainerStarted","Data":"a7a974edcb966fa8b2e41e3dfae9e2d2a2b294fbfd9f757b8112962b7561c9fe"} Oct 30 21:45:01.819570 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:01.819494 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d68156-29889945-2nc2b" event={"ID":"30fdce00-7aeb-4812-a580-8dde48515851","Type":"ContainerStarted","Data":"6b18a371cc0579afa521d73823d30cfc272f6719d4dc5b1f5a103b0d37e40643"} Oct 30 21:45:01.819779 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:01.819762 5748 scope.go:122] "RemoveContainer" containerID="a7a974edcb966fa8b2e41e3dfae9e2d2a2b294fbfd9f757b8112962b7561c9fe" Oct 30 21:45:01.821883 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:01.821803 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91874e-29889945-hbl8j" event={"ID":"dba2ae0f-a9ff-48da-b2e6-f8011473d615","Type":"ContainerStarted","Data":"281af2d728f7afac284a058c5e571ef209f0af2913ff8ea33fed465698e9d6d3"} Oct 30 21:45:01.821883 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:01.821850 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91874e-29889945-hbl8j" event={"ID":"dba2ae0f-a9ff-48da-b2e6-f8011473d615","Type":"ContainerStarted","Data":"4b47561863418238acca62312e1a17d4942c1beab182479336b1ba90e0477fb9"} Oct 30 21:45:01.821883 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:01.821880 5748 scope.go:122] "RemoveContainer" containerID="281af2d728f7afac284a058c5e571ef209f0af2913ff8ea33fed465698e9d6d3" Oct 30 21:45:01.827881 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:01.824415 5748 generic.go:356] "Generic (PLEG): container finished" podID="28a0c85c-d54f-42f9-8c57-1508fa1b2c82" containerID="7fa2d1a51ce4f208cf08eb5bac5f182621c011de993c160418c86e0fb41e98fd" exitCode=60 Oct 30 21:45:01.827881 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:01.824533 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889945-6lzds" event={"ID":"28a0c85c-d54f-42f9-8c57-1508fa1b2c82","Type":"ContainerDied","Data":"7fa2d1a51ce4f208cf08eb5bac5f182621c011de993c160418c86e0fb41e98fd"} Oct 30 21:45:01.827881 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:01.824569 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889945-6lzds" event={"ID":"28a0c85c-d54f-42f9-8c57-1508fa1b2c82","Type":"ContainerStarted","Data":"e52d9c1d65ab42163336a142295af1a0ac6a72c36ba4016bf753a79f63b74881"} Oct 30 21:45:01.827881 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:01.824679 5748 scope.go:122] "RemoveContainer" containerID="7fa2d1a51ce4f208cf08eb5bac5f182621c011de993c160418c86e0fb41e98fd" Oct 30 21:45:01.920455 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:01.920362 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w9njw"] Oct 30 21:45:01.920706 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:01.920676 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w9njw" podUID="eddd7c6a-a409-468c-831a-2e3944e8059a" containerName="istio-proxy" containerID="cri-o://28e30ab5b8b860f6025b5609190944393492289c5c824f209d025908dc449e5a" gracePeriod=30 Oct 30 21:45:01.937550 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:01.937488 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-691598-29889945-vzhl4" podStartSLOduration=1.937469874 podStartE2EDuration="1.937469874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:01.935953075 +0000 UTC m=+1285.678474011" watchObservedRunningTime="2026-10-30 21:45:01.937469874 +0000 UTC m=+1285.679990811" Oct 30 21:45:02.007185 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:02.007159 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d69ca2e6-dx6dj" Oct 30 21:45:02.056584 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:02.056523 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2977c254-smh4c"] Oct 30 21:45:02.066734 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:02.066703 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2977c254-smh4c"] Oct 30 21:45:02.147368 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:02.147337 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe58c836-3dbd-4389-83c8-f66b58152106-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe58c836-3dbd-4389-83c8-f66b58152106-maas-api-openshift-ca\") pod \"fe58c836-3dbd-4389-83c8-f66b58152106\" (UID: \"fe58c836-3dbd-4389-83c8-f66b58152106\") " Oct 30 21:45:02.147516 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:02.147445 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe58c836-3dbd-4389-83c8-f66b58152106-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe58c836-3dbd-4389-83c8-f66b58152106-maas-api-xks-ca\") pod \"fe58c836-3dbd-4389-83c8-f66b58152106\" (UID: \"fe58c836-3dbd-4389-83c8-f66b58152106\") " Oct 30 21:45:02.147595 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:02.147586 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe58c836-3dbd-4389-83c8-f66b58152106-kube-api-access-t7prq\" (UniqueName: \"kubernetes.io/projected/fe58c836-3dbd-4389-83c8-f66b58152106-kube-api-access-t7prq\") pod \"fe58c836-3dbd-4389-83c8-f66b58152106\" (UID: \"fe58c836-3dbd-4389-83c8-f66b58152106\") " Oct 30 21:45:02.147698 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:02.147674 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe58c836-3dbd-4389-83c8-f66b58152106-maas-api-openshift-ca" pod "fe58c836-3dbd-4389-83c8-f66b58152106" (UID: "fe58c836-3dbd-4389-83c8-f66b58152106"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:02.148058 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:02.148039 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe58c836-3dbd-4389-83c8-f66b58152106-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:02.201369 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:02.201335 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe58c836-3dbd-4389-83c8-f66b58152106-kube-api-access-t7prq" pod "fe58c836-3dbd-4389-83c8-f66b58152106" (UID: "fe58c836-3dbd-4389-83c8-f66b58152106"). InnerVolumeSpecName "kube-api-access-t7prq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:02.201369 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:02.201351 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe58c836-3dbd-4389-83c8-f66b58152106-maas-api-xks-ca" pod "fe58c836-3dbd-4389-83c8-f66b58152106" (UID: "fe58c836-3dbd-4389-83c8-f66b58152106"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:45:02.249641 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:02.249612 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe58c836-3dbd-4389-83c8-f66b58152106-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:02.249641 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:02.249637 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7prq\" (UniqueName: \"kubernetes.io/projected/fe58c836-3dbd-4389-83c8-f66b58152106-kube-api-access-t7prq\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:02.422003 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:02.421976 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae34ed3b-a844-4f73-bb27-4c75ba279377" path="/var/lib/kubelet/pods/ae34ed3b-a844-4f73-bb27-4c75ba279377/volumes" Oct 30 21:45:02.833818 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:02.833785 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889945-6lzds" event={"ID":"28a0c85c-d54f-42f9-8c57-1508fa1b2c82","Type":"ContainerStarted","Data":"1c68f6c08c762a73f7e4480dfbae471cb953fc6f28e91786da6bf43daf175e19"} Oct 30 21:45:02.835363 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:02.835337 5748 generic.go:356] "Generic (PLEG): container finished" podID="c042e96b-236e-419e-b531-43798f9bb25f" containerID="71d37b7c49c78323e7d722a811ec6105096d9e4ac24adb0bada22bc8e0bbd8e3" exitCode=60 Oct 30 21:45:02.835471 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:02.835418 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-691598-29889945-vzhl4" event={"ID":"c042e96b-236e-419e-b531-43798f9bb25f","Type":"ContainerDied","Data":"71d37b7c49c78323e7d722a811ec6105096d9e4ac24adb0bada22bc8e0bbd8e3"} Oct 30 21:45:02.835608 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:02.835593 5748 scope.go:122] "RemoveContainer" containerID="71d37b7c49c78323e7d722a811ec6105096d9e4ac24adb0bada22bc8e0bbd8e3" Oct 30 21:45:02.836931 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:02.836907 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d69ca2e6-dx6dj" event={"ID":"fe58c836-3dbd-4389-83c8-f66b58152106","Type":"ContainerDied","Data":"17150878b0cbab22fb763bfa1b896f3e0416780fd7776dcc2fc073b099fc0a4e"} Oct 30 21:45:02.837038 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:02.836955 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17150878b0cbab22fb763bfa1b896f3e0416780fd7776dcc2fc073b099fc0a4e" Oct 30 21:45:02.837038 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:02.836918 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d69ca2e6-dx6dj" Oct 30 21:45:02.839663 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:02.839640 5748 generic.go:356] "Generic (PLEG): container finished" podID="30fdce00-7aeb-4812-a580-8dde48515851" containerID="a7a974edcb966fa8b2e41e3dfae9e2d2a2b294fbfd9f757b8112962b7561c9fe" exitCode=60 Oct 30 21:45:02.839663 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:02.839663 5748 generic.go:356] "Generic (PLEG): container finished" podID="30fdce00-7aeb-4812-a580-8dde48515851" containerID="4e099e3d8725a46911addfe55675d67264679e0a59a62769239a715a00066676" exitCode=60 Oct 30 21:45:02.839797 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:02.839720 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d68156-29889945-2nc2b" event={"ID":"30fdce00-7aeb-4812-a580-8dde48515851","Type":"ContainerDied","Data":"a7a974edcb966fa8b2e41e3dfae9e2d2a2b294fbfd9f757b8112962b7561c9fe"} Oct 30 21:45:02.839797 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:02.839775 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d68156-29889945-2nc2b" event={"ID":"30fdce00-7aeb-4812-a580-8dde48515851","Type":"ContainerDied","Data":"4e099e3d8725a46911addfe55675d67264679e0a59a62769239a715a00066676"} Oct 30 21:45:02.839917 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:02.839803 5748 scope.go:122] "RemoveContainer" containerID="a7a974edcb966fa8b2e41e3dfae9e2d2a2b294fbfd9f757b8112962b7561c9fe" Oct 30 21:45:02.840035 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:02.840019 5748 scope.go:122] "RemoveContainer" containerID="4e099e3d8725a46911addfe55675d67264679e0a59a62769239a715a00066676" Oct 30 21:45:02.840300 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:45:02.840273 5748 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-d68156-29889945-2nc2b_odh-ai-gateway-infra(30fdce00-7aeb-4812-a580-8dde48515851)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d68156-29889945-2nc2b" podUID="30fdce00-7aeb-4812-a580-8dde48515851" Oct 30 21:45:02.841646 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:02.841624 5748 generic.go:356] "Generic (PLEG): container finished" podID="dba2ae0f-a9ff-48da-b2e6-f8011473d615" containerID="281af2d728f7afac284a058c5e571ef209f0af2913ff8ea33fed465698e9d6d3" exitCode=60 Oct 30 21:45:02.841766 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:02.841701 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91874e-29889945-hbl8j" event={"ID":"dba2ae0f-a9ff-48da-b2e6-f8011473d615","Type":"ContainerDied","Data":"281af2d728f7afac284a058c5e571ef209f0af2913ff8ea33fed465698e9d6d3"} Oct 30 21:45:02.841766 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:02.841733 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91874e-29889945-hbl8j" event={"ID":"dba2ae0f-a9ff-48da-b2e6-f8011473d615","Type":"ContainerStarted","Data":"405baa727e42cbd0fc7d9cbfec6b47b4a3491465b31567cc76aec2eb76cd0ecc"} Oct 30 21:45:02.880124 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:02.880108 5748 scope.go:122] "RemoveContainer" containerID="a7a974edcb966fa8b2e41e3dfae9e2d2a2b294fbfd9f757b8112962b7561c9fe" Oct 30 21:45:02.880428 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:45:02.880409 5748 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7a974edcb966fa8b2e41e3dfae9e2d2a2b294fbfd9f757b8112962b7561c9fe\": container with ID starting with a7a974edcb966fa8b2e41e3dfae9e2d2a2b294fbfd9f757b8112962b7561c9fe not found: ID does not exist" containerID="a7a974edcb966fa8b2e41e3dfae9e2d2a2b294fbfd9f757b8112962b7561c9fe" Oct 30 21:45:02.880474 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:02.880438 5748 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7a974edcb966fa8b2e41e3dfae9e2d2a2b294fbfd9f757b8112962b7561c9fe"} err="failed to get container status \"a7a974edcb966fa8b2e41e3dfae9e2d2a2b294fbfd9f757b8112962b7561c9fe\": rpc error: code = NotFound desc = could not find container \"a7a974edcb966fa8b2e41e3dfae9e2d2a2b294fbfd9f757b8112962b7561c9fe\": container with ID starting with a7a974edcb966fa8b2e41e3dfae9e2d2a2b294fbfd9f757b8112962b7561c9fe not found: ID does not exist" Oct 30 21:45:02.951704 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:02.951656 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889945-6lzds" podStartSLOduration=2.951642014 podStartE2EDuration="2.951642014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:02.950418806 +0000 UTC m=+1286.692939739" watchObservedRunningTime="2026-10-30 21:45:02.951642014 +0000 UTC m=+1286.694162949" Oct 30 21:45:03.357668 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:03.357610 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91874e-29889945-hbl8j" podStartSLOduration=3.357596196 podStartE2EDuration="3.357596196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:03.354584331 +0000 UTC m=+1287.097105268" watchObservedRunningTime="2026-10-30 21:45:03.357596196 +0000 UTC m=+1287.100117156" Oct 30 21:45:03.850441 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:03.850358 5748 generic.go:356] "Generic (PLEG): container finished" podID="28a0c85c-d54f-42f9-8c57-1508fa1b2c82" containerID="1c68f6c08c762a73f7e4480dfbae471cb953fc6f28e91786da6bf43daf175e19" exitCode=60 Oct 30 21:45:03.850858 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:03.850442 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889945-6lzds" event={"ID":"28a0c85c-d54f-42f9-8c57-1508fa1b2c82","Type":"ContainerDied","Data":"1c68f6c08c762a73f7e4480dfbae471cb953fc6f28e91786da6bf43daf175e19"} Oct 30 21:45:03.850858 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:03.850523 5748 scope.go:122] "RemoveContainer" containerID="7fa2d1a51ce4f208cf08eb5bac5f182621c011de993c160418c86e0fb41e98fd" Oct 30 21:45:03.850858 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:03.850726 5748 scope.go:122] "RemoveContainer" containerID="1c68f6c08c762a73f7e4480dfbae471cb953fc6f28e91786da6bf43daf175e19" Oct 30 21:45:03.851121 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:45:03.851097 5748 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889945-6lzds_odh-ai-gateway-infra(28a0c85c-d54f-42f9-8c57-1508fa1b2c82)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889945-6lzds" podUID="28a0c85c-d54f-42f9-8c57-1508fa1b2c82" Oct 30 21:45:03.852487 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:03.852466 5748 generic.go:356] "Generic (PLEG): container finished" podID="c042e96b-236e-419e-b531-43798f9bb25f" containerID="e5bf67f97340cd037ff2871e8d7bc729214a2e1494fb3f544c4f1787f016bcb9" exitCode=60 Oct 30 21:45:03.852577 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:03.852550 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-691598-29889945-vzhl4" event={"ID":"c042e96b-236e-419e-b531-43798f9bb25f","Type":"ContainerDied","Data":"e5bf67f97340cd037ff2871e8d7bc729214a2e1494fb3f544c4f1787f016bcb9"} Oct 30 21:45:03.852776 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:03.852761 5748 scope.go:122] "RemoveContainer" containerID="e5bf67f97340cd037ff2871e8d7bc729214a2e1494fb3f544c4f1787f016bcb9" Oct 30 21:45:03.853100 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:45:03.853075 5748 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-691598-29889945-vzhl4_odh-ai-gateway-infra(c042e96b-236e-419e-b531-43798f9bb25f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-691598-29889945-vzhl4" podUID="c042e96b-236e-419e-b531-43798f9bb25f" Oct 30 21:45:03.854339 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:03.854322 5748 scope.go:122] "RemoveContainer" containerID="4e099e3d8725a46911addfe55675d67264679e0a59a62769239a715a00066676" Oct 30 21:45:03.854595 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:45:03.854577 5748 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-d68156-29889945-2nc2b_odh-ai-gateway-infra(30fdce00-7aeb-4812-a580-8dde48515851)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d68156-29889945-2nc2b" podUID="30fdce00-7aeb-4812-a580-8dde48515851" Oct 30 21:45:03.855967 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:03.855942 5748 generic.go:356] "Generic (PLEG): container finished" podID="dba2ae0f-a9ff-48da-b2e6-f8011473d615" containerID="405baa727e42cbd0fc7d9cbfec6b47b4a3491465b31567cc76aec2eb76cd0ecc" exitCode=60 Oct 30 21:45:03.856041 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:03.855977 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91874e-29889945-hbl8j" event={"ID":"dba2ae0f-a9ff-48da-b2e6-f8011473d615","Type":"ContainerDied","Data":"405baa727e42cbd0fc7d9cbfec6b47b4a3491465b31567cc76aec2eb76cd0ecc"} Oct 30 21:45:03.856252 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:03.856237 5748 scope.go:122] "RemoveContainer" containerID="405baa727e42cbd0fc7d9cbfec6b47b4a3491465b31567cc76aec2eb76cd0ecc" Oct 30 21:45:03.856571 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:45:03.856549 5748 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-91874e-29889945-hbl8j_odh-ai-gateway-infra(dba2ae0f-a9ff-48da-b2e6-f8011473d615)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91874e-29889945-hbl8j" podUID="dba2ae0f-a9ff-48da-b2e6-f8011473d615" Oct 30 21:45:03.866806 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:03.866786 5748 scope.go:122] "RemoveContainer" containerID="71d37b7c49c78323e7d722a811ec6105096d9e4ac24adb0bada22bc8e0bbd8e3" Oct 30 21:45:03.881043 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:03.881020 5748 scope.go:122] "RemoveContainer" containerID="281af2d728f7afac284a058c5e571ef209f0af2913ff8ea33fed465698e9d6d3" Oct 30 21:45:04.049104 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:04.049074 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b6b39cc-kjgl2"] Oct 30 21:45:04.053020 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:04.052993 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b6b39cc-kjgl2"] Oct 30 21:45:04.409417 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:04.409377 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-962109ad-8km6j"] Oct 30 21:45:04.410064 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:04.410043 5748 cpu_manager.go:414] "containerMap: removing container" podUID="fe58c836-3dbd-4389-83c8-f66b58152106" containerName="revoke-keys" Oct 30 21:45:04.410185 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:04.410063 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe58c836-3dbd-4389-83c8-f66b58152106" containerName="revoke-keys" Oct 30 21:45:04.410244 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:04.410233 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe58c836-3dbd-4389-83c8-f66b58152106" containerName="revoke-keys" Oct 30 21:45:04.415015 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:04.414989 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-962109ad-8km6j" Oct 30 21:45:04.424600 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:04.424576 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f3aa9b8-02e6-4e1d-856d-7e6507d6c220" path="/var/lib/kubelet/pods/5f3aa9b8-02e6-4e1d-856d-7e6507d6c220/volumes" Oct 30 21:45:04.468272 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:04.468247 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:45:04.472930 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:04.472882 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-962109ad-8km6j"] Oct 30 21:45:04.475876 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:04.475852 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkmqv\" (UniqueName: \"kubernetes.io/projected/6c22b3be-2a12-46e0-8465-a319ddede7e4-kube-api-access-mkmqv\") pod \"maas-api-revoke-sub-962109ad-8km6j\" (UID: \"6c22b3be-2a12-46e0-8465-a319ddede7e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-962109ad-8km6j" Oct 30 21:45:04.476034 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:04.476013 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c22b3be-2a12-46e0-8465-a319ddede7e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-962109ad-8km6j\" (UID: \"6c22b3be-2a12-46e0-8465-a319ddede7e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-962109ad-8km6j" Oct 30 21:45:04.476273 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:04.476251 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c22b3be-2a12-46e0-8465-a319ddede7e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-962109ad-8km6j\" (UID: \"6c22b3be-2a12-46e0-8465-a319ddede7e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-962109ad-8km6j" Oct 30 21:45:04.577341 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:04.577297 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkmqv\" (UniqueName: \"kubernetes.io/projected/6c22b3be-2a12-46e0-8465-a319ddede7e4-kube-api-access-mkmqv\") pod \"maas-api-revoke-sub-962109ad-8km6j\" (UID: \"6c22b3be-2a12-46e0-8465-a319ddede7e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-962109ad-8km6j" Oct 30 21:45:04.577534 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:04.577404 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c22b3be-2a12-46e0-8465-a319ddede7e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-962109ad-8km6j\" (UID: \"6c22b3be-2a12-46e0-8465-a319ddede7e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-962109ad-8km6j" Oct 30 21:45:04.577608 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:04.577570 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c22b3be-2a12-46e0-8465-a319ddede7e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-962109ad-8km6j\" (UID: \"6c22b3be-2a12-46e0-8465-a319ddede7e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-962109ad-8km6j" Oct 30 21:45:04.578340 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:04.578314 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c22b3be-2a12-46e0-8465-a319ddede7e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-962109ad-8km6j\" (UID: \"6c22b3be-2a12-46e0-8465-a319ddede7e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-962109ad-8km6j" Oct 30 21:45:04.579884 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:04.579862 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c22b3be-2a12-46e0-8465-a319ddede7e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-962109ad-8km6j\" (UID: \"6c22b3be-2a12-46e0-8465-a319ddede7e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-962109ad-8km6j" Oct 30 21:45:04.639241 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:04.639200 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkmqv\" (UniqueName: \"kubernetes.io/projected/6c22b3be-2a12-46e0-8465-a319ddede7e4-kube-api-access-mkmqv\") pod \"maas-api-revoke-sub-962109ad-8km6j\" (UID: \"6c22b3be-2a12-46e0-8465-a319ddede7e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-962109ad-8km6j" Oct 30 21:45:04.729586 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:04.729496 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-962109ad-8km6j" Oct 30 21:45:04.868117 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:04.868083 5748 scope.go:122] "RemoveContainer" containerID="1c68f6c08c762a73f7e4480dfbae471cb953fc6f28e91786da6bf43daf175e19" Oct 30 21:45:04.868499 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:45:04.868468 5748 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889945-6lzds_odh-ai-gateway-infra(28a0c85c-d54f-42f9-8c57-1508fa1b2c82)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889945-6lzds" podUID="28a0c85c-d54f-42f9-8c57-1508fa1b2c82" Oct 30 21:45:04.871286 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:04.871259 5748 scope.go:122] "RemoveContainer" containerID="e5bf67f97340cd037ff2871e8d7bc729214a2e1494fb3f544c4f1787f016bcb9" Oct 30 21:45:04.871598 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:45:04.871573 5748 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-691598-29889945-vzhl4_odh-ai-gateway-infra(c042e96b-236e-419e-b531-43798f9bb25f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-691598-29889945-vzhl4" podUID="c042e96b-236e-419e-b531-43798f9bb25f" Oct 30 21:45:04.873547 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:04.873525 5748 scope.go:122] "RemoveContainer" containerID="405baa727e42cbd0fc7d9cbfec6b47b4a3491465b31567cc76aec2eb76cd0ecc" Oct 30 21:45:04.873832 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:45:04.873808 5748 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-91874e-29889945-hbl8j_odh-ai-gateway-infra(dba2ae0f-a9ff-48da-b2e6-f8011473d615)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91874e-29889945-hbl8j" podUID="dba2ae0f-a9ff-48da-b2e6-f8011473d615" Oct 30 21:45:05.120508 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:45:05.120479 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c22b3be_2a12_46e0_8465_a319ddede7e4.slice/crio-dc2b3779c82f75a8ec075ca3ec2599a763fdd0d53abc1ad289d87aebd049b6ec WatchSource:0}: Error finding container dc2b3779c82f75a8ec075ca3ec2599a763fdd0d53abc1ad289d87aebd049b6ec: Status 404 returned error can't find the container with id dc2b3779c82f75a8ec075ca3ec2599a763fdd0d53abc1ad289d87aebd049b6ec Oct 30 21:45:05.120652 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:05.120586 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-962109ad-8km6j"] Oct 30 21:45:05.583957 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:05.583927 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dde10fdf-j8rq6"] Oct 30 21:45:05.588210 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:05.588193 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde10fdf-j8rq6" Oct 30 21:45:05.646774 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:05.646743 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dde10fdf-j8rq6"] Oct 30 21:45:05.693011 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:05.692928 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkq7g\" (UniqueName: \"kubernetes.io/projected/fab2565b-8b9b-4e45-baa2-6ec1f5cf002f-kube-api-access-qkq7g\") pod \"maas-api-revoke-sub-dde10fdf-j8rq6\" (UID: \"fab2565b-8b9b-4e45-baa2-6ec1f5cf002f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde10fdf-j8rq6" Oct 30 21:45:05.693140 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:05.693047 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fab2565b-8b9b-4e45-baa2-6ec1f5cf002f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dde10fdf-j8rq6\" (UID: \"fab2565b-8b9b-4e45-baa2-6ec1f5cf002f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde10fdf-j8rq6" Oct 30 21:45:05.693195 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:05.693139 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fab2565b-8b9b-4e45-baa2-6ec1f5cf002f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dde10fdf-j8rq6\" (UID: \"fab2565b-8b9b-4e45-baa2-6ec1f5cf002f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde10fdf-j8rq6" Oct 30 21:45:05.794552 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:05.794513 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkq7g\" (UniqueName: \"kubernetes.io/projected/fab2565b-8b9b-4e45-baa2-6ec1f5cf002f-kube-api-access-qkq7g\") pod \"maas-api-revoke-sub-dde10fdf-j8rq6\" (UID: \"fab2565b-8b9b-4e45-baa2-6ec1f5cf002f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde10fdf-j8rq6" Oct 30 21:45:05.794715 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:05.794641 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fab2565b-8b9b-4e45-baa2-6ec1f5cf002f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dde10fdf-j8rq6\" (UID: \"fab2565b-8b9b-4e45-baa2-6ec1f5cf002f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde10fdf-j8rq6" Oct 30 21:45:05.794715 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:05.794705 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fab2565b-8b9b-4e45-baa2-6ec1f5cf002f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dde10fdf-j8rq6\" (UID: \"fab2565b-8b9b-4e45-baa2-6ec1f5cf002f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde10fdf-j8rq6" Oct 30 21:45:05.795413 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:05.795386 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fab2565b-8b9b-4e45-baa2-6ec1f5cf002f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dde10fdf-j8rq6\" (UID: \"fab2565b-8b9b-4e45-baa2-6ec1f5cf002f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde10fdf-j8rq6" Oct 30 21:45:05.796989 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:05.796969 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fab2565b-8b9b-4e45-baa2-6ec1f5cf002f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dde10fdf-j8rq6\" (UID: \"fab2565b-8b9b-4e45-baa2-6ec1f5cf002f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde10fdf-j8rq6" Oct 30 21:45:05.856731 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:05.856699 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkq7g\" (UniqueName: \"kubernetes.io/projected/fab2565b-8b9b-4e45-baa2-6ec1f5cf002f-kube-api-access-qkq7g\") pod \"maas-api-revoke-sub-dde10fdf-j8rq6\" (UID: \"fab2565b-8b9b-4e45-baa2-6ec1f5cf002f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde10fdf-j8rq6" Oct 30 21:45:05.882724 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:05.882692 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-962109ad-8km6j" event={"ID":"6c22b3be-2a12-46e0-8465-a319ddede7e4","Type":"ContainerStarted","Data":"947069212c057fa8ee43d12161b6cc54b0b9c89b062ad7fcafa6570c517d0f38"} Oct 30 21:45:05.883094 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:05.882748 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-962109ad-8km6j" event={"ID":"6c22b3be-2a12-46e0-8465-a319ddede7e4","Type":"ContainerStarted","Data":"dc2b3779c82f75a8ec075ca3ec2599a763fdd0d53abc1ad289d87aebd049b6ec"} Oct 30 21:45:05.902186 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:05.902158 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde10fdf-j8rq6" Oct 30 21:45:06.005948 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:06.003998 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-962109ad-8km6j" podStartSLOduration=2.003978989 podStartE2EDuration="2.003978989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:06.001426104 +0000 UTC m=+1289.743947040" watchObservedRunningTime="2026-10-30 21:45:06.003978989 +0000 UTC m=+1289.746499925" Oct 30 21:45:06.303966 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:06.302271 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dde10fdf-j8rq6"] Oct 30 21:45:06.309342 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:45:06.309309 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfab2565b_8b9b_4e45_baa2_6ec1f5cf002f.slice/crio-d96cccb19033c29734c4a24080b50ebcf5cded7d6a1f56c61bb41ba67711f3fd WatchSource:0}: Error finding container d96cccb19033c29734c4a24080b50ebcf5cded7d6a1f56c61bb41ba67711f3fd: Status 404 returned error can't find the container with id d96cccb19033c29734c4a24080b50ebcf5cded7d6a1f56c61bb41ba67711f3fd Oct 30 21:45:06.938956 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:06.938704 5748 generic.go:356] "Generic (PLEG): container finished" podID="6c22b3be-2a12-46e0-8465-a319ddede7e4" containerID="947069212c057fa8ee43d12161b6cc54b0b9c89b062ad7fcafa6570c517d0f38" exitCode=0 Oct 30 21:45:06.938956 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:06.938854 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-962109ad-8km6j" event={"ID":"6c22b3be-2a12-46e0-8465-a319ddede7e4","Type":"ContainerDied","Data":"947069212c057fa8ee43d12161b6cc54b0b9c89b062ad7fcafa6570c517d0f38"} Oct 30 21:45:06.971102 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:06.965397 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde10fdf-j8rq6" event={"ID":"fab2565b-8b9b-4e45-baa2-6ec1f5cf002f","Type":"ContainerStarted","Data":"0912051ed1016479be605d73ee12dc91683398351045e88f99a83f322cb6f417"} Oct 30 21:45:06.971102 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:06.965538 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde10fdf-j8rq6" event={"ID":"fab2565b-8b9b-4e45-baa2-6ec1f5cf002f","Type":"ContainerStarted","Data":"d96cccb19033c29734c4a24080b50ebcf5cded7d6a1f56c61bb41ba67711f3fd"} Oct 30 21:45:07.290377 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:07.290239 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde10fdf-j8rq6" podStartSLOduration=2.290220845 podStartE2EDuration="2.290220845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:07.221813907 +0000 UTC m=+1290.964334845" watchObservedRunningTime="2026-10-30 21:45:07.290220845 +0000 UTC m=+1291.032741777" Oct 30 21:45:07.898315 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:07.860650 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q"] Oct 30 21:45:07.898315 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:07.870303 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q" Oct 30 21:45:07.947035 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:07.944283 5748 patch_prober.go:29] interesting pod/e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:45:07.947035 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:07.944346 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7" podUID="d2cdb6d5-0628-435d-ade1-e0aa84440718" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:07.987218 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:07.971724 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxr5d\" (UniqueName: \"kubernetes.io/projected/00245d73-3768-4ecd-9ddb-371c69b05e56-kube-api-access-kxr5d\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q\" (UID: \"00245d73-3768-4ecd-9ddb-371c69b05e56\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q" Oct 30 21:45:07.987218 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:07.971788 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00245d73-3768-4ecd-9ddb-371c69b05e56-istio-data\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q\" (UID: \"00245d73-3768-4ecd-9ddb-371c69b05e56\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q" Oct 30 21:45:07.987218 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:07.971861 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00245d73-3768-4ecd-9ddb-371c69b05e56-istio-podinfo\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q\" (UID: \"00245d73-3768-4ecd-9ddb-371c69b05e56\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q" Oct 30 21:45:07.987218 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:07.971888 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00245d73-3768-4ecd-9ddb-371c69b05e56-istio-token\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q\" (UID: \"00245d73-3768-4ecd-9ddb-371c69b05e56\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q" Oct 30 21:45:07.987218 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:07.971938 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00245d73-3768-4ecd-9ddb-371c69b05e56-workload-socket\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q\" (UID: \"00245d73-3768-4ecd-9ddb-371c69b05e56\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q" Oct 30 21:45:07.987218 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:07.971962 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00245d73-3768-4ecd-9ddb-371c69b05e56-istio-envoy\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q\" (UID: \"00245d73-3768-4ecd-9ddb-371c69b05e56\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q" Oct 30 21:45:07.987218 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:07.972161 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00245d73-3768-4ecd-9ddb-371c69b05e56-istiod-ca-cert\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q\" (UID: \"00245d73-3768-4ecd-9ddb-371c69b05e56\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q" Oct 30 21:45:07.987218 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:07.972195 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00245d73-3768-4ecd-9ddb-371c69b05e56-credential-socket\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q\" (UID: \"00245d73-3768-4ecd-9ddb-371c69b05e56\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q" Oct 30 21:45:07.987218 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:07.972232 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00245d73-3768-4ecd-9ddb-371c69b05e56-workload-certs\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q\" (UID: \"00245d73-3768-4ecd-9ddb-371c69b05e56\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q" Oct 30 21:45:08.037570 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:08.033975 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9"] Oct 30 21:45:08.070926 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:08.049616 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-69e4b2/simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj" podUID="698fc0f2-64ce-4e7e-b482-8ebd9f33f84d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:08.099179 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:08.085810 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00245d73-3768-4ecd-9ddb-371c69b05e56-workload-certs\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q\" (UID: \"00245d73-3768-4ecd-9ddb-371c69b05e56\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q" Oct 30 21:45:08.099541 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:08.086240 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00245d73-3768-4ecd-9ddb-371c69b05e56-workload-certs\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q\" (UID: \"00245d73-3768-4ecd-9ddb-371c69b05e56\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q" Oct 30 21:45:08.100059 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:08.099705 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxr5d\" (UniqueName: \"kubernetes.io/projected/00245d73-3768-4ecd-9ddb-371c69b05e56-kube-api-access-kxr5d\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q\" (UID: \"00245d73-3768-4ecd-9ddb-371c69b05e56\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q" Oct 30 21:45:08.100311 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:08.100291 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00245d73-3768-4ecd-9ddb-371c69b05e56-istio-data\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q\" (UID: \"00245d73-3768-4ecd-9ddb-371c69b05e56\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q" Oct 30 21:45:08.100868 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:08.100843 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00245d73-3768-4ecd-9ddb-371c69b05e56-istio-podinfo\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q\" (UID: \"00245d73-3768-4ecd-9ddb-371c69b05e56\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q" Oct 30 21:45:08.115993 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:08.100706 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00245d73-3768-4ecd-9ddb-371c69b05e56-istio-data\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q\" (UID: \"00245d73-3768-4ecd-9ddb-371c69b05e56\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q" Oct 30 21:45:08.115993 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:08.101663 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9" Oct 30 21:45:08.116925 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:08.101717 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00245d73-3768-4ecd-9ddb-371c69b05e56-istio-token\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q\" (UID: \"00245d73-3768-4ecd-9ddb-371c69b05e56\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q" Oct 30 21:45:08.117128 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:08.117105 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00245d73-3768-4ecd-9ddb-371c69b05e56-workload-socket\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q\" (UID: \"00245d73-3768-4ecd-9ddb-371c69b05e56\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q" Oct 30 21:45:08.117260 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:08.117244 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00245d73-3768-4ecd-9ddb-371c69b05e56-istio-envoy\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q\" (UID: \"00245d73-3768-4ecd-9ddb-371c69b05e56\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q" Oct 30 21:45:08.124919 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:08.117604 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00245d73-3768-4ecd-9ddb-371c69b05e56-istiod-ca-cert\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q\" (UID: \"00245d73-3768-4ecd-9ddb-371c69b05e56\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q" Oct 30 21:45:08.137019 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:08.136982 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00245d73-3768-4ecd-9ddb-371c69b05e56-credential-socket\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q\" (UID: \"00245d73-3768-4ecd-9ddb-371c69b05e56\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q" Oct 30 21:45:08.137346 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:08.135670 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00245d73-3768-4ecd-9ddb-371c69b05e56-istiod-ca-cert\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q\" (UID: \"00245d73-3768-4ecd-9ddb-371c69b05e56\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q" Oct 30 21:45:08.138206 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:08.125662 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00245d73-3768-4ecd-9ddb-371c69b05e56-workload-socket\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q\" (UID: \"00245d73-3768-4ecd-9ddb-371c69b05e56\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q" Oct 30 21:45:08.160274 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:08.138621 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00245d73-3768-4ecd-9ddb-371c69b05e56-credential-socket\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q\" (UID: \"00245d73-3768-4ecd-9ddb-371c69b05e56\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q" Oct 30 21:45:08.212923 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:08.200974 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9"] Oct 30 21:45:08.253957 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:08.252102 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmjvp\" (UniqueName: \"kubernetes.io/projected/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-kube-api-access-wmjvp\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9\" (UID: \"0018c42f-45f5-4417-8b83-6d0cd9bd3c02\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9" Oct 30 21:45:08.253957 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:08.252276 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-istio-data\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9\" (UID: \"0018c42f-45f5-4417-8b83-6d0cd9bd3c02\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9" Oct 30 21:45:08.253957 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:08.252364 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-credential-socket\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9\" (UID: \"0018c42f-45f5-4417-8b83-6d0cd9bd3c02\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9" Oct 30 21:45:08.253957 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:08.252428 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-istiod-ca-cert\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9\" (UID: \"0018c42f-45f5-4417-8b83-6d0cd9bd3c02\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9" Oct 30 21:45:08.253957 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:08.252562 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-istio-podinfo\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9\" (UID: \"0018c42f-45f5-4417-8b83-6d0cd9bd3c02\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9" Oct 30 21:45:08.253957 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:08.252595 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-workload-socket\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9\" (UID: \"0018c42f-45f5-4417-8b83-6d0cd9bd3c02\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9" Oct 30 21:45:08.253957 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:08.252711 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-workload-certs\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9\" (UID: \"0018c42f-45f5-4417-8b83-6d0cd9bd3c02\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9" Oct 30 21:45:08.253957 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:08.252737 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-istio-envoy\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9\" (UID: \"0018c42f-45f5-4417-8b83-6d0cd9bd3c02\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9" Oct 30 21:45:08.255975 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:08.255942 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-istio-token\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9\" (UID: \"0018c42f-45f5-4417-8b83-6d0cd9bd3c02\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9" Oct 30 21:45:08.297623 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:08.297565 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q"] Oct 30 21:45:08.313952 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:08.300749 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00245d73-3768-4ecd-9ddb-371c69b05e56-istio-token\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q\" (UID: \"00245d73-3768-4ecd-9ddb-371c69b05e56\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q" Oct 30 21:45:08.313952 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:08.304290 5748 generic.go:356] "Generic (PLEG): container finished" podID="eddd7c6a-a409-468c-831a-2e3944e8059a" containerID="28e30ab5b8b860f6025b5609190944393492289c5c824f209d025908dc449e5a" exitCode=0 Oct 30 21:45:08.313952 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:08.304702 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w9njw" event={"ID":"eddd7c6a-a409-468c-831a-2e3944e8059a","Type":"ContainerDied","Data":"28e30ab5b8b860f6025b5609190944393492289c5c824f209d025908dc449e5a"} Oct 30 21:45:08.313952 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:08.305469 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00245d73-3768-4ecd-9ddb-371c69b05e56-istio-envoy\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q\" (UID: \"00245d73-3768-4ecd-9ddb-371c69b05e56\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q" Oct 30 21:45:08.313952 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:08.306173 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00245d73-3768-4ecd-9ddb-371c69b05e56-istio-podinfo\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q\" (UID: \"00245d73-3768-4ecd-9ddb-371c69b05e56\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q" Oct 30 21:45:08.368922 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:08.356625 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxr5d\" (UniqueName: \"kubernetes.io/projected/00245d73-3768-4ecd-9ddb-371c69b05e56-kube-api-access-kxr5d\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q\" (UID: \"00245d73-3768-4ecd-9ddb-371c69b05e56\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q" Oct 30 21:45:08.368922 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:08.357515 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-workload-certs\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9\" (UID: \"0018c42f-45f5-4417-8b83-6d0cd9bd3c02\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9" Oct 30 21:45:08.368922 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:08.357556 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-istio-envoy\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9\" (UID: \"0018c42f-45f5-4417-8b83-6d0cd9bd3c02\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9" Oct 30 21:45:08.368922 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:08.357587 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-istio-token\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9\" (UID: \"0018c42f-45f5-4417-8b83-6d0cd9bd3c02\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9" Oct 30 21:45:08.368922 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:08.357703 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmjvp\" (UniqueName: \"kubernetes.io/projected/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-kube-api-access-wmjvp\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9\" (UID: \"0018c42f-45f5-4417-8b83-6d0cd9bd3c02\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9" Oct 30 21:45:08.368922 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:08.357797 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-istio-data\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9\" (UID: \"0018c42f-45f5-4417-8b83-6d0cd9bd3c02\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9" Oct 30 21:45:08.368922 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:08.357851 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-credential-socket\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9\" (UID: \"0018c42f-45f5-4417-8b83-6d0cd9bd3c02\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9" Oct 30 21:45:08.368922 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:08.357910 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-istiod-ca-cert\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9\" (UID: \"0018c42f-45f5-4417-8b83-6d0cd9bd3c02\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9" Oct 30 21:45:08.368922 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:08.357994 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-istio-podinfo\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9\" (UID: \"0018c42f-45f5-4417-8b83-6d0cd9bd3c02\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9" Oct 30 21:45:08.368922 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:08.358024 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-workload-socket\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9\" (UID: \"0018c42f-45f5-4417-8b83-6d0cd9bd3c02\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9" Oct 30 21:45:08.368922 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:08.358516 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-workload-socket\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9\" (UID: \"0018c42f-45f5-4417-8b83-6d0cd9bd3c02\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9" Oct 30 21:45:08.368922 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:08.358795 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-workload-certs\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9\" (UID: \"0018c42f-45f5-4417-8b83-6d0cd9bd3c02\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9" Oct 30 21:45:08.368922 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:08.360547 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-istio-data\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9\" (UID: \"0018c42f-45f5-4417-8b83-6d0cd9bd3c02\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9" Oct 30 21:45:08.368922 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:08.368585 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-istiod-ca-cert\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9\" (UID: \"0018c42f-45f5-4417-8b83-6d0cd9bd3c02\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9" Oct 30 21:45:08.368922 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:08.368876 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-credential-socket\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9\" (UID: \"0018c42f-45f5-4417-8b83-6d0cd9bd3c02\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9" Oct 30 21:45:08.614973 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:08.500514 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-istio-podinfo\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9\" (UID: \"0018c42f-45f5-4417-8b83-6d0cd9bd3c02\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9" Oct 30 21:45:08.614973 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:08.530368 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-istio-envoy\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9\" (UID: \"0018c42f-45f5-4417-8b83-6d0cd9bd3c02\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9" Oct 30 21:45:08.614973 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:08.537137 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q" Oct 30 21:45:08.614973 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:08.603972 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmjvp\" (UniqueName: \"kubernetes.io/projected/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-kube-api-access-wmjvp\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9\" (UID: \"0018c42f-45f5-4417-8b83-6d0cd9bd3c02\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9" Oct 30 21:45:08.680961 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:08.679996 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d68156/distinct-w3-d68156-kserve-565894fdb4-85t7z" podUID="13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:08.728097 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:08.706386 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d68156/simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk" podUID="db285e9c-ca87-43fa-8f06-69e271e5c2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:08.728097 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:08.708439 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-istio-token\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9\" (UID: \"0018c42f-45f5-4417-8b83-6d0cd9bd3c02\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9" Oct 30 21:45:08.920038 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:08.863798 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1de75a/simulator-w4-1de75a-kserve-6887dd4b46-kc5c2" podUID="926fa847-0c3c-43e0-b378-04c82433cd98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Oct 30 21:45:08.920038 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:08.870205 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9" Oct 30 21:45:08.920038 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:08.899884 5748 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h" podUID="0a0b0425-2959-4690-9847-38852e167230" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": context deadline exceeded" Oct 30 21:45:08.920038 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:08.900951 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h" podUID="0a0b0425-2959-4690-9847-38852e167230" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:08.987413 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:08.959196 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ea374/distinct-2-w1-6ea374-kserve-68fb49889-hpg2k" podUID="ab533734-e0d8-4acb-854d-12851a87bc58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 30 21:45:08.987413 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:08.959381 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c7d477/distinct-w5-c7d477-kserve-554d7f974d-6zw6s" podUID="09d39cc0-b68d-48fd-918c-7d2518f384aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:09.080929 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:09.074069 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-691598/simulator-w0-691598-kserve-76fb5d5bc7-l5gqk" podUID="4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:09.112923 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:09.110673 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91874e/distinct-w2-91874e-kserve-58d6bdfc75-bdp5z" podUID="cb39b415-f406-47bd-b591-be7e09240f5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:09.160867 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:09.155036 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-69e4b2/simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj" podUID="698fc0f2-64ce-4e7e-b482-8ebd9f33f84d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:09.160867 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:09.155475 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-69e4b2/simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj" Oct 30 21:45:09.347184 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:09.288871 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ea374/simulator-w1-6ea374-kserve-775cfdf98b-p5q25" podUID="b0d53207-65f3-4d43-baba-66bc4c366885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 21:45:09.525989 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:09.525949 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-69e4b2/premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b" podUID="3ed43145-33bd-4c11-bb29-dc852511f6d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 21:45:09.617633 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:09.617559 5748 generic.go:356] "Generic (PLEG): container finished" podID="fab2565b-8b9b-4e45-baa2-6ec1f5cf002f" containerID="0912051ed1016479be605d73ee12dc91683398351045e88f99a83f322cb6f417" exitCode=0 Oct 30 21:45:09.620793 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:09.620767 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde10fdf-j8rq6" event={"ID":"fab2565b-8b9b-4e45-baa2-6ec1f5cf002f","Type":"ContainerDied","Data":"0912051ed1016479be605d73ee12dc91683398351045e88f99a83f322cb6f417"} Oct 30 21:45:09.644511 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:09.644483 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-69e4b2/simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj" Oct 30 21:45:09.650387 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:09.650163 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w9njw" Oct 30 21:45:09.678920 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:09.673088 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-962109ad-8km6j" Oct 30 21:45:09.713930 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:09.713030 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d68156/simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk" podUID="db285e9c-ca87-43fa-8f06-69e271e5c2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 30 21:45:09.713930 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:09.713568 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d68156/simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk" Oct 30 21:45:09.852103 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:09.849120 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c22b3be-2a12-46e0-8465-a319ddede7e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c22b3be-2a12-46e0-8465-a319ddede7e4-maas-api-xks-ca\") pod \"6c22b3be-2a12-46e0-8465-a319ddede7e4\" (UID: \"6c22b3be-2a12-46e0-8465-a319ddede7e4\") " Oct 30 21:45:09.852103 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:09.849223 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eddd7c6a-a409-468c-831a-2e3944e8059a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eddd7c6a-a409-468c-831a-2e3944e8059a-istio-podinfo\") pod \"eddd7c6a-a409-468c-831a-2e3944e8059a\" (UID: \"eddd7c6a-a409-468c-831a-2e3944e8059a\") " Oct 30 21:45:09.852103 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:09.849256 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c22b3be-2a12-46e0-8465-a319ddede7e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c22b3be-2a12-46e0-8465-a319ddede7e4-maas-api-openshift-ca\") pod \"6c22b3be-2a12-46e0-8465-a319ddede7e4\" (UID: \"6c22b3be-2a12-46e0-8465-a319ddede7e4\") " Oct 30 21:45:09.852103 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:09.849288 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eddd7c6a-a409-468c-831a-2e3944e8059a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eddd7c6a-a409-468c-831a-2e3944e8059a-istio-data\") pod \"eddd7c6a-a409-468c-831a-2e3944e8059a\" (UID: \"eddd7c6a-a409-468c-831a-2e3944e8059a\") " Oct 30 21:45:09.852103 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:09.849320 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c22b3be-2a12-46e0-8465-a319ddede7e4-kube-api-access-mkmqv\" (UniqueName: \"kubernetes.io/projected/6c22b3be-2a12-46e0-8465-a319ddede7e4-kube-api-access-mkmqv\") pod \"6c22b3be-2a12-46e0-8465-a319ddede7e4\" (UID: \"6c22b3be-2a12-46e0-8465-a319ddede7e4\") " Oct 30 21:45:09.852103 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:09.849377 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eddd7c6a-a409-468c-831a-2e3944e8059a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eddd7c6a-a409-468c-831a-2e3944e8059a-credential-socket\") pod \"eddd7c6a-a409-468c-831a-2e3944e8059a\" (UID: \"eddd7c6a-a409-468c-831a-2e3944e8059a\") " Oct 30 21:45:09.852103 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:09.849492 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eddd7c6a-a409-468c-831a-2e3944e8059a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eddd7c6a-a409-468c-831a-2e3944e8059a-istio-envoy\") pod \"eddd7c6a-a409-468c-831a-2e3944e8059a\" (UID: \"eddd7c6a-a409-468c-831a-2e3944e8059a\") " Oct 30 21:45:09.852103 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:09.849521 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eddd7c6a-a409-468c-831a-2e3944e8059a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eddd7c6a-a409-468c-831a-2e3944e8059a-istiod-ca-cert\") pod \"eddd7c6a-a409-468c-831a-2e3944e8059a\" (UID: \"eddd7c6a-a409-468c-831a-2e3944e8059a\") " Oct 30 21:45:09.852103 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:09.849646 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eddd7c6a-a409-468c-831a-2e3944e8059a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eddd7c6a-a409-468c-831a-2e3944e8059a-workload-certs\") pod \"eddd7c6a-a409-468c-831a-2e3944e8059a\" (UID: \"eddd7c6a-a409-468c-831a-2e3944e8059a\") " Oct 30 21:45:09.944922 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:09.944671 5748 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-w9njw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:45:09.944922 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:09.944734 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w9njw" podUID="eddd7c6a-a409-468c-831a-2e3944e8059a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:09.949670 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:09.949635 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eddd7c6a-a409-468c-831a-2e3944e8059a-credential-socket" pod "eddd7c6a-a409-468c-831a-2e3944e8059a" (UID: "eddd7c6a-a409-468c-831a-2e3944e8059a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:45:09.961921 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:09.956473 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h" podUID="0a0b0425-2959-4690-9847-38852e167230" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Oct 30 21:45:09.961921 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:09.956852 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h" Oct 30 21:45:09.970384 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:09.966110 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c22b3be-2a12-46e0-8465-a319ddede7e4-maas-api-openshift-ca" pod "6c22b3be-2a12-46e0-8465-a319ddede7e4" (UID: "6c22b3be-2a12-46e0-8465-a319ddede7e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:09.971017 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:09.970693 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eddd7c6a-a409-468c-831a-2e3944e8059a-istiod-ca-cert" pod "eddd7c6a-a409-468c-831a-2e3944e8059a" (UID: "eddd7c6a-a409-468c-831a-2e3944e8059a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:09.977794 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:09.976983 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ea374/distinct-2-w1-6ea374-kserve-68fb49889-hpg2k" podUID="ab533734-e0d8-4acb-854d-12851a87bc58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:09.979909 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:09.979789 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eddd7c6a-a409-468c-831a-2e3944e8059a-istio-envoy" pod "eddd7c6a-a409-468c-831a-2e3944e8059a" (UID: "eddd7c6a-a409-468c-831a-2e3944e8059a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:45:09.981311 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:09.980499 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6ea374/distinct-2-w1-6ea374-kserve-68fb49889-hpg2k" Oct 30 21:45:09.986194 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:09.984146 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eddd7c6a-a409-468c-831a-2e3944e8059a-workload-certs" pod "eddd7c6a-a409-468c-831a-2e3944e8059a" (UID: "eddd7c6a-a409-468c-831a-2e3944e8059a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:45:10.011127 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:10.011060 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eddd7c6a-a409-468c-831a-2e3944e8059a-istio-podinfo" pod "eddd7c6a-a409-468c-831a-2e3944e8059a" (UID: "eddd7c6a-a409-468c-831a-2e3944e8059a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:45:10.022741 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:10.020992 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91874e/distinct-2-w2-91874e-kserve-69fcf64f44-qm96r" podUID="0cb16f33-b208-4c11-bb12-322829bd51e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:10.096960 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:10.034916 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eddd7c6a-a409-468c-831a-2e3944e8059a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eddd7c6a-a409-468c-831a-2e3944e8059a-workload-socket\") pod \"eddd7c6a-a409-468c-831a-2e3944e8059a\" (UID: \"eddd7c6a-a409-468c-831a-2e3944e8059a\") " Oct 30 21:45:10.096960 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:10.035079 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eddd7c6a-a409-468c-831a-2e3944e8059a-kube-api-access-hrbvm\" (UniqueName: \"kubernetes.io/projected/eddd7c6a-a409-468c-831a-2e3944e8059a-kube-api-access-hrbvm\") pod \"eddd7c6a-a409-468c-831a-2e3944e8059a\" (UID: \"eddd7c6a-a409-468c-831a-2e3944e8059a\") " Oct 30 21:45:10.096960 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:10.035122 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eddd7c6a-a409-468c-831a-2e3944e8059a-istio-token\" (UniqueName: \"kubernetes.io/projected/eddd7c6a-a409-468c-831a-2e3944e8059a-istio-token\") pod \"eddd7c6a-a409-468c-831a-2e3944e8059a\" (UID: \"eddd7c6a-a409-468c-831a-2e3944e8059a\") " Oct 30 21:45:10.096960 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:10.035521 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eddd7c6a-a409-468c-831a-2e3944e8059a-workload-socket" pod "eddd7c6a-a409-468c-831a-2e3944e8059a" (UID: "eddd7c6a-a409-468c-831a-2e3944e8059a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:45:10.096960 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:10.046478 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d68156/simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk" Oct 30 21:45:10.096960 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:10.046986 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eddd7c6a-a409-468c-831a-2e3944e8059a-istio-data" pod "eddd7c6a-a409-468c-831a-2e3944e8059a" (UID: "eddd7c6a-a409-468c-831a-2e3944e8059a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:45:10.139996 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:10.108764 5748 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eddd7c6a-a409-468c-831a-2e3944e8059a-istio-envoy\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:10.139996 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:10.108795 5748 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eddd7c6a-a409-468c-831a-2e3944e8059a-istiod-ca-cert\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:10.139996 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:10.108840 5748 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eddd7c6a-a409-468c-831a-2e3944e8059a-workload-certs\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:10.139996 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:10.108856 5748 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eddd7c6a-a409-468c-831a-2e3944e8059a-workload-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:10.139996 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:10.108910 5748 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eddd7c6a-a409-468c-831a-2e3944e8059a-istio-podinfo\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:10.139996 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:10.108926 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c22b3be-2a12-46e0-8465-a319ddede7e4-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:10.139996 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:10.108965 5748 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eddd7c6a-a409-468c-831a-2e3944e8059a-istio-data\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:10.139996 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:10.108981 5748 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eddd7c6a-a409-468c-831a-2e3944e8059a-credential-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:10.146144 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:10.146105 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eddd7c6a-a409-468c-831a-2e3944e8059a-istio-token" pod "eddd7c6a-a409-468c-831a-2e3944e8059a" (UID: "eddd7c6a-a409-468c-831a-2e3944e8059a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:10.218336 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:10.194168 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:45:10.218336 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:10.194258 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:45:10.218336 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:10.194304 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:45:10.316292 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:10.241401 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eddd7c6a-a409-468c-831a-2e3944e8059a-kube-api-access-hrbvm" pod "eddd7c6a-a409-468c-831a-2e3944e8059a" (UID: "eddd7c6a-a409-468c-831a-2e3944e8059a"). InnerVolumeSpecName "kube-api-access-hrbvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:10.316292 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:10.241492 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c22b3be-2a12-46e0-8465-a319ddede7e4-maas-api-xks-ca" pod "6c22b3be-2a12-46e0-8465-a319ddede7e4" (UID: "6c22b3be-2a12-46e0-8465-a319ddede7e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:45:10.316292 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:10.288769 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c22b3be-2a12-46e0-8465-a319ddede7e4-kube-api-access-mkmqv" pod "6c22b3be-2a12-46e0-8465-a319ddede7e4" (UID: "6c22b3be-2a12-46e0-8465-a319ddede7e4"). InnerVolumeSpecName "kube-api-access-mkmqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:10.316292 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:10.295086 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h" Oct 30 21:45:10.316292 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:10.301171 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q"] Oct 30 21:45:10.317858 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:10.317485 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrbvm\" (UniqueName: \"kubernetes.io/projected/eddd7c6a-a409-468c-831a-2e3944e8059a-kube-api-access-hrbvm\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:10.317858 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:10.317518 5748 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eddd7c6a-a409-468c-831a-2e3944e8059a-istio-token\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:10.317858 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:10.317535 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c22b3be-2a12-46e0-8465-a319ddede7e4-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:10.317858 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:10.317565 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkmqv\" (UniqueName: \"kubernetes.io/projected/6c22b3be-2a12-46e0-8465-a319ddede7e4-kube-api-access-mkmqv\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:10.327927 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:10.327828 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6ea374/distinct-2-w1-6ea374-kserve-68fb49889-hpg2k" Oct 30 21:45:10.429290 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:10.429235 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:45:10.429444 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:10.429390 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:45:10.429515 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:10.429480 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:45:10.470750 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:10.470681 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9"] Oct 30 21:45:10.640739 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:10.640700 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q" event={"ID":"00245d73-3768-4ecd-9ddb-371c69b05e56","Type":"ContainerStarted","Data":"c37162bb04d8c3387b63a6bde42678db1be3569fb39897e7d750a707127aaa38"} Oct 30 21:45:10.641689 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:10.641660 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q" event={"ID":"00245d73-3768-4ecd-9ddb-371c69b05e56","Type":"ContainerStarted","Data":"0e4aa4645c1b2c12aad5d2ca55ca8851ee625b03b6f29c9bb03dc32509b025de"} Oct 30 21:45:10.642764 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:10.642739 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9" event={"ID":"0018c42f-45f5-4417-8b83-6d0cd9bd3c02","Type":"ContainerStarted","Data":"7d391d65d81a93ecd71d4e52ff50c7edb2affe5ae50dbdb5c65d46108bbbd095"} Oct 30 21:45:10.642979 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:10.642960 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9" event={"ID":"0018c42f-45f5-4417-8b83-6d0cd9bd3c02","Type":"ContainerStarted","Data":"1874e21e53fc2bfa6a88e0fcb7bef4f71c305315f4f8e51b7d85e9bb89498e1d"} Oct 30 21:45:10.644978 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:10.644960 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w9njw" Oct 30 21:45:10.646173 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:10.645028 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w9njw" event={"ID":"eddd7c6a-a409-468c-831a-2e3944e8059a","Type":"ContainerDied","Data":"3818df89a2954358a1a830aaf348b9ddf1b467e0f42677d93526e4fa75e33431"} Oct 30 21:45:10.646342 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:10.646326 5748 scope.go:122] "RemoveContainer" containerID="28e30ab5b8b860f6025b5609190944393492289c5c824f209d025908dc449e5a" Oct 30 21:45:10.648386 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:10.646522 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-962109ad-8km6j" Oct 30 21:45:10.648937 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:10.646551 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-962109ad-8km6j" event={"ID":"6c22b3be-2a12-46e0-8465-a319ddede7e4","Type":"ContainerDied","Data":"dc2b3779c82f75a8ec075ca3ec2599a763fdd0d53abc1ad289d87aebd049b6ec"} Oct 30 21:45:10.649082 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:10.649068 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc2b3779c82f75a8ec075ca3ec2599a763fdd0d53abc1ad289d87aebd049b6ec" Oct 30 21:45:10.826822 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:10.826784 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde10fdf-j8rq6" Oct 30 21:45:10.884438 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:10.884363 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9" podStartSLOduration=3.884341492 podStartE2EDuration="3.884341492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:10.879310169 +0000 UTC m=+1294.621831104" watchObservedRunningTime="2026-10-30 21:45:10.884341492 +0000 UTC m=+1294.626862430" Oct 30 21:45:10.894335 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:10.894304 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9" Oct 30 21:45:10.946024 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:10.945994 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fab2565b-8b9b-4e45-baa2-6ec1f5cf002f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fab2565b-8b9b-4e45-baa2-6ec1f5cf002f-maas-api-openshift-ca\") pod \"fab2565b-8b9b-4e45-baa2-6ec1f5cf002f\" (UID: \"fab2565b-8b9b-4e45-baa2-6ec1f5cf002f\") " Oct 30 21:45:10.946295 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:10.946274 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fab2565b-8b9b-4e45-baa2-6ec1f5cf002f-kube-api-access-qkq7g\" (UniqueName: \"kubernetes.io/projected/fab2565b-8b9b-4e45-baa2-6ec1f5cf002f-kube-api-access-qkq7g\") pod \"fab2565b-8b9b-4e45-baa2-6ec1f5cf002f\" (UID: \"fab2565b-8b9b-4e45-baa2-6ec1f5cf002f\") " Oct 30 21:45:10.946339 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:10.946321 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fab2565b-8b9b-4e45-baa2-6ec1f5cf002f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fab2565b-8b9b-4e45-baa2-6ec1f5cf002f-maas-api-xks-ca\") pod \"fab2565b-8b9b-4e45-baa2-6ec1f5cf002f\" (UID: \"fab2565b-8b9b-4e45-baa2-6ec1f5cf002f\") " Oct 30 21:45:10.946393 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:10.946375 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fab2565b-8b9b-4e45-baa2-6ec1f5cf002f-maas-api-openshift-ca" pod "fab2565b-8b9b-4e45-baa2-6ec1f5cf002f" (UID: "fab2565b-8b9b-4e45-baa2-6ec1f5cf002f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:10.946843 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:10.946817 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fab2565b-8b9b-4e45-baa2-6ec1f5cf002f-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:10.948672 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:10.948633 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fab2565b-8b9b-4e45-baa2-6ec1f5cf002f-kube-api-access-qkq7g" pod "fab2565b-8b9b-4e45-baa2-6ec1f5cf002f" (UID: "fab2565b-8b9b-4e45-baa2-6ec1f5cf002f"). InnerVolumeSpecName "kube-api-access-qkq7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:10.948765 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:10.948674 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fab2565b-8b9b-4e45-baa2-6ec1f5cf002f-maas-api-xks-ca" pod "fab2565b-8b9b-4e45-baa2-6ec1f5cf002f" (UID: "fab2565b-8b9b-4e45-baa2-6ec1f5cf002f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:45:11.050020 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:11.049143 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkq7g\" (UniqueName: \"kubernetes.io/projected/fab2565b-8b9b-4e45-baa2-6ec1f5cf002f-kube-api-access-qkq7g\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:11.050020 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:11.049184 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fab2565b-8b9b-4e45-baa2-6ec1f5cf002f-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:11.065822 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:11.062626 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60fb5a05-gdrnv"] Oct 30 21:45:11.065822 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:11.063640 5748 cpu_manager.go:414] "containerMap: removing container" podUID="6c22b3be-2a12-46e0-8465-a319ddede7e4" containerName="revoke-keys" Oct 30 21:45:11.065822 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:11.063659 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c22b3be-2a12-46e0-8465-a319ddede7e4" containerName="revoke-keys" Oct 30 21:45:11.065822 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:11.063674 5748 cpu_manager.go:414] "containerMap: removing container" podUID="fab2565b-8b9b-4e45-baa2-6ec1f5cf002f" containerName="revoke-keys" Oct 30 21:45:11.065822 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:11.063683 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fab2565b-8b9b-4e45-baa2-6ec1f5cf002f" containerName="revoke-keys" Oct 30 21:45:11.065822 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:11.063719 5748 cpu_manager.go:414] "containerMap: removing container" podUID="eddd7c6a-a409-468c-831a-2e3944e8059a" containerName="istio-proxy" Oct 30 21:45:11.065822 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:11.063727 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eddd7c6a-a409-468c-831a-2e3944e8059a" containerName="istio-proxy" Oct 30 21:45:11.065822 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:11.063872 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c22b3be-2a12-46e0-8465-a319ddede7e4" containerName="revoke-keys" Oct 30 21:45:11.065822 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:11.063886 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="fab2565b-8b9b-4e45-baa2-6ec1f5cf002f" containerName="revoke-keys" Oct 30 21:45:11.065822 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:11.063913 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="eddd7c6a-a409-468c-831a-2e3944e8059a" containerName="istio-proxy" Oct 30 21:45:11.067311 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:11.067234 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q" podStartSLOduration=4.067216588 podStartE2EDuration="4.067216588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:11.055873025 +0000 UTC m=+1294.798393959" watchObservedRunningTime="2026-10-30 21:45:11.067216588 +0000 UTC m=+1294.809737523" Oct 30 21:45:11.072139 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:11.072111 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60fb5a05-gdrnv" Oct 30 21:45:11.080053 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:11.080026 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4ff1ba1-j2gkx"] Oct 30 21:45:11.085432 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:11.085403 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4ff1ba1-j2gkx"] Oct 30 21:45:11.148922 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:11.143250 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60fb5a05-gdrnv"] Oct 30 21:45:11.150998 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:11.150911 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8fd0e2c-8495-4f2a-9d0e-166200ef8d59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60fb5a05-gdrnv\" (UID: \"c8fd0e2c-8495-4f2a-9d0e-166200ef8d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60fb5a05-gdrnv" Oct 30 21:45:11.151325 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:11.151303 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8fd0e2c-8495-4f2a-9d0e-166200ef8d59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60fb5a05-gdrnv\" (UID: \"c8fd0e2c-8495-4f2a-9d0e-166200ef8d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60fb5a05-gdrnv" Oct 30 21:45:11.151501 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:11.151485 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wkwd\" (UniqueName: \"kubernetes.io/projected/c8fd0e2c-8495-4f2a-9d0e-166200ef8d59-kube-api-access-8wkwd\") pod \"maas-api-revoke-sub-60fb5a05-gdrnv\" (UID: \"c8fd0e2c-8495-4f2a-9d0e-166200ef8d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60fb5a05-gdrnv" Oct 30 21:45:11.185732 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:11.185691 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w9njw"] Oct 30 21:45:11.191517 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:11.191332 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w9njw"] Oct 30 21:45:11.258092 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:11.256953 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8fd0e2c-8495-4f2a-9d0e-166200ef8d59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60fb5a05-gdrnv\" (UID: \"c8fd0e2c-8495-4f2a-9d0e-166200ef8d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60fb5a05-gdrnv" Oct 30 21:45:11.258092 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:11.257169 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8fd0e2c-8495-4f2a-9d0e-166200ef8d59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60fb5a05-gdrnv\" (UID: \"c8fd0e2c-8495-4f2a-9d0e-166200ef8d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60fb5a05-gdrnv" Oct 30 21:45:11.258092 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:11.257252 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wkwd\" (UniqueName: \"kubernetes.io/projected/c8fd0e2c-8495-4f2a-9d0e-166200ef8d59-kube-api-access-8wkwd\") pod \"maas-api-revoke-sub-60fb5a05-gdrnv\" (UID: \"c8fd0e2c-8495-4f2a-9d0e-166200ef8d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60fb5a05-gdrnv" Oct 30 21:45:11.258853 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:11.258807 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8fd0e2c-8495-4f2a-9d0e-166200ef8d59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60fb5a05-gdrnv\" (UID: \"c8fd0e2c-8495-4f2a-9d0e-166200ef8d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60fb5a05-gdrnv" Oct 30 21:45:11.262182 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:11.262122 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8fd0e2c-8495-4f2a-9d0e-166200ef8d59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60fb5a05-gdrnv\" (UID: \"c8fd0e2c-8495-4f2a-9d0e-166200ef8d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60fb5a05-gdrnv" Oct 30 21:45:11.344973 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:11.339133 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wkwd\" (UniqueName: \"kubernetes.io/projected/c8fd0e2c-8495-4f2a-9d0e-166200ef8d59-kube-api-access-8wkwd\") pod \"maas-api-revoke-sub-60fb5a05-gdrnv\" (UID: \"c8fd0e2c-8495-4f2a-9d0e-166200ef8d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60fb5a05-gdrnv" Oct 30 21:45:11.392274 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:11.392233 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60fb5a05-gdrnv" Oct 30 21:45:11.548060 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:11.547959 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q" Oct 30 21:45:11.553920 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:11.549559 5748 patch_prober.go:29] interesting pod/e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 21:45:11.553920 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:11.549619 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q" podUID="00245d73-3768-4ecd-9ddb-371c69b05e56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 21:45:11.696630 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:11.696593 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde10fdf-j8rq6" event={"ID":"fab2565b-8b9b-4e45-baa2-6ec1f5cf002f","Type":"ContainerDied","Data":"d96cccb19033c29734c4a24080b50ebcf5cded7d6a1f56c61bb41ba67711f3fd"} Oct 30 21:45:11.696892 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:11.696873 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d96cccb19033c29734c4a24080b50ebcf5cded7d6a1f56c61bb41ba67711f3fd" Oct 30 21:45:11.697082 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:11.697068 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde10fdf-j8rq6" Oct 30 21:45:11.698709 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:11.698679 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e843d27-d4nx9"] Oct 30 21:45:11.735919 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:11.713263 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e843d27-d4nx9" Oct 30 21:45:11.769050 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:11.769012 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e843d27-d4nx9"] Oct 30 21:45:11.777233 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:11.777204 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz2ft\" (UniqueName: \"kubernetes.io/projected/59ccc507-2625-4679-b3b8-b0796c1fd01a-kube-api-access-zz2ft\") pod \"maas-api-revoke-sub-6e843d27-d4nx9\" (UID: \"59ccc507-2625-4679-b3b8-b0796c1fd01a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e843d27-d4nx9" Oct 30 21:45:11.777564 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:11.777548 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59ccc507-2625-4679-b3b8-b0796c1fd01a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e843d27-d4nx9\" (UID: \"59ccc507-2625-4679-b3b8-b0796c1fd01a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e843d27-d4nx9" Oct 30 21:45:11.777684 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:11.777674 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59ccc507-2625-4679-b3b8-b0796c1fd01a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e843d27-d4nx9\" (UID: \"59ccc507-2625-4679-b3b8-b0796c1fd01a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e843d27-d4nx9" Oct 30 21:45:11.879455 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:11.879413 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59ccc507-2625-4679-b3b8-b0796c1fd01a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e843d27-d4nx9\" (UID: \"59ccc507-2625-4679-b3b8-b0796c1fd01a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e843d27-d4nx9" Oct 30 21:45:11.879650 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:11.879468 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59ccc507-2625-4679-b3b8-b0796c1fd01a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e843d27-d4nx9\" (UID: \"59ccc507-2625-4679-b3b8-b0796c1fd01a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e843d27-d4nx9" Oct 30 21:45:11.879650 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:11.879567 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz2ft\" (UniqueName: \"kubernetes.io/projected/59ccc507-2625-4679-b3b8-b0796c1fd01a-kube-api-access-zz2ft\") pod \"maas-api-revoke-sub-6e843d27-d4nx9\" (UID: \"59ccc507-2625-4679-b3b8-b0796c1fd01a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e843d27-d4nx9" Oct 30 21:45:11.880079 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:11.880058 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59ccc507-2625-4679-b3b8-b0796c1fd01a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e843d27-d4nx9\" (UID: \"59ccc507-2625-4679-b3b8-b0796c1fd01a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e843d27-d4nx9" Oct 30 21:45:11.881971 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:11.881955 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59ccc507-2625-4679-b3b8-b0796c1fd01a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e843d27-d4nx9\" (UID: \"59ccc507-2625-4679-b3b8-b0796c1fd01a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e843d27-d4nx9" Oct 30 21:45:11.896149 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:11.896121 5748 patch_prober.go:29] interesting pod/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 21:45:11.896310 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:11.896193 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9" podUID="0018c42f-45f5-4417-8b83-6d0cd9bd3c02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 21:45:11.943973 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:11.943093 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60fb5a05-gdrnv"] Oct 30 21:45:11.957925 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:11.957639 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz2ft\" (UniqueName: \"kubernetes.io/projected/59ccc507-2625-4679-b3b8-b0796c1fd01a-kube-api-access-zz2ft\") pod \"maas-api-revoke-sub-6e843d27-d4nx9\" (UID: \"59ccc507-2625-4679-b3b8-b0796c1fd01a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e843d27-d4nx9" Oct 30 21:45:12.092394 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:12.092361 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e843d27-d4nx9" Oct 30 21:45:12.458460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:12.458415 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f7e0cdb-2273-4bfa-bba8-fa5c86f13f5d" path="/var/lib/kubelet/pods/5f7e0cdb-2273-4bfa-bba8-fa5c86f13f5d/volumes" Oct 30 21:45:12.459026 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:12.459007 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eddd7c6a-a409-468c-831a-2e3944e8059a" path="/var/lib/kubelet/pods/eddd7c6a-a409-468c-831a-2e3944e8059a/volumes" Oct 30 21:45:12.556820 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:12.556148 5748 patch_prober.go:29] interesting pod/e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 21:45:12.556820 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:12.556227 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q" podUID="00245d73-3768-4ecd-9ddb-371c69b05e56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 21:45:12.588361 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:12.588339 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e843d27-d4nx9"] Oct 30 21:45:12.595296 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:45:12.595270 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59ccc507_2625_4679_b3b8_b0796c1fd01a.slice/crio-e6615c5da5c21c55da025bb7e9a6ffe27397a06e0f61ddfeefdbab12bf846382 WatchSource:0}: Error finding container e6615c5da5c21c55da025bb7e9a6ffe27397a06e0f61ddfeefdbab12bf846382: Status 404 returned error can't find the container with id e6615c5da5c21c55da025bb7e9a6ffe27397a06e0f61ddfeefdbab12bf846382 Oct 30 21:45:12.762072 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:12.762034 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e843d27-d4nx9" event={"ID":"59ccc507-2625-4679-b3b8-b0796c1fd01a","Type":"ContainerStarted","Data":"e6615c5da5c21c55da025bb7e9a6ffe27397a06e0f61ddfeefdbab12bf846382"} Oct 30 21:45:12.764205 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:12.764179 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60fb5a05-gdrnv" event={"ID":"c8fd0e2c-8495-4f2a-9d0e-166200ef8d59","Type":"ContainerStarted","Data":"e6d38f666087febb45357f2d5db5ff2bf1a371758662834f9c82de68a5e54a69"} Oct 30 21:45:12.764278 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:12.764223 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60fb5a05-gdrnv" event={"ID":"c8fd0e2c-8495-4f2a-9d0e-166200ef8d59","Type":"ContainerStarted","Data":"b2f65032d1fd12e15779f5097e380f845e6daf851026a6b568ea0c85661f4701"} Oct 30 21:45:12.895275 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:12.895243 5748 patch_prober.go:29] interesting pod/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 21:45:12.895480 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:12.895294 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9" podUID="0018c42f-45f5-4417-8b83-6d0cd9bd3c02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 21:45:12.968956 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:12.965919 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60fb5a05-gdrnv" podStartSLOduration=1.965796028 podStartE2EDuration="1.965796028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:12.964151044 +0000 UTC m=+1296.706671984" watchObservedRunningTime="2026-10-30 21:45:12.965796028 +0000 UTC m=+1296.708316964" Oct 30 21:45:13.546404 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:13.541346 5748 patch_prober.go:29] interesting pod/e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 21:45:13.546404 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:13.541440 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q" podUID="00245d73-3768-4ecd-9ddb-371c69b05e56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 21:45:13.773237 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:13.773190 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e843d27-d4nx9" event={"ID":"59ccc507-2625-4679-b3b8-b0796c1fd01a","Type":"ContainerStarted","Data":"ad0be915e62d41afba3069e592010cdadfc6fcd178d09742b76c95b4c284d3f0"} Oct 30 21:45:13.774786 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:13.774752 5748 generic.go:356] "Generic (PLEG): container finished" podID="c8fd0e2c-8495-4f2a-9d0e-166200ef8d59" containerID="e6d38f666087febb45357f2d5db5ff2bf1a371758662834f9c82de68a5e54a69" exitCode=0 Oct 30 21:45:13.774916 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:13.774833 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60fb5a05-gdrnv" event={"ID":"c8fd0e2c-8495-4f2a-9d0e-166200ef8d59","Type":"ContainerDied","Data":"e6d38f666087febb45357f2d5db5ff2bf1a371758662834f9c82de68a5e54a69"} Oct 30 21:45:13.894368 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:13.894335 5748 patch_prober.go:29] interesting pod/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 21:45:13.894550 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:13.894390 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9" podUID="0018c42f-45f5-4417-8b83-6d0cd9bd3c02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 21:45:13.922923 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:13.917425 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e843d27-d4nx9" podStartSLOduration=2.9174025649999997 podStartE2EDuration="2.917402565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:13.898720437 +0000 UTC m=+1297.641241375" watchObservedRunningTime="2026-10-30 21:45:13.917402565 +0000 UTC m=+1297.659923500" Oct 30 21:45:14.458583 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:14.431696 5748 scope.go:122] "RemoveContainer" containerID="4e099e3d8725a46911addfe55675d67264679e0a59a62769239a715a00066676" Oct 30 21:45:14.552923 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:14.549558 5748 patch_prober.go:29] interesting pod/e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 21:45:14.552923 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:14.549636 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q" podUID="00245d73-3768-4ecd-9ddb-371c69b05e56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 21:45:14.785474 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:14.785361 5748 generic.go:356] "Generic (PLEG): container finished" podID="59ccc507-2625-4679-b3b8-b0796c1fd01a" containerID="ad0be915e62d41afba3069e592010cdadfc6fcd178d09742b76c95b4c284d3f0" exitCode=0 Oct 30 21:45:14.785474 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:14.785451 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e843d27-d4nx9" event={"ID":"59ccc507-2625-4679-b3b8-b0796c1fd01a","Type":"ContainerDied","Data":"ad0be915e62d41afba3069e592010cdadfc6fcd178d09742b76c95b4c284d3f0"} Oct 30 21:45:14.895118 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:14.895079 5748 patch_prober.go:29] interesting pod/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 21:45:14.895337 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:14.895309 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9" podUID="0018c42f-45f5-4417-8b83-6d0cd9bd3c02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 21:45:15.260861 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:15.260833 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60fb5a05-gdrnv" Oct 30 21:45:15.394684 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:15.394650 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8fd0e2c-8495-4f2a-9d0e-166200ef8d59-kube-api-access-8wkwd\" (UniqueName: \"kubernetes.io/projected/c8fd0e2c-8495-4f2a-9d0e-166200ef8d59-kube-api-access-8wkwd\") pod \"c8fd0e2c-8495-4f2a-9d0e-166200ef8d59\" (UID: \"c8fd0e2c-8495-4f2a-9d0e-166200ef8d59\") " Oct 30 21:45:15.395061 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:15.395037 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8fd0e2c-8495-4f2a-9d0e-166200ef8d59-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8fd0e2c-8495-4f2a-9d0e-166200ef8d59-maas-api-xks-ca\") pod \"c8fd0e2c-8495-4f2a-9d0e-166200ef8d59\" (UID: \"c8fd0e2c-8495-4f2a-9d0e-166200ef8d59\") " Oct 30 21:45:15.395312 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:15.395295 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8fd0e2c-8495-4f2a-9d0e-166200ef8d59-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8fd0e2c-8495-4f2a-9d0e-166200ef8d59-maas-api-openshift-ca\") pod \"c8fd0e2c-8495-4f2a-9d0e-166200ef8d59\" (UID: \"c8fd0e2c-8495-4f2a-9d0e-166200ef8d59\") " Oct 30 21:45:15.396452 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:15.395725 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8fd0e2c-8495-4f2a-9d0e-166200ef8d59-maas-api-openshift-ca" pod "c8fd0e2c-8495-4f2a-9d0e-166200ef8d59" (UID: "c8fd0e2c-8495-4f2a-9d0e-166200ef8d59"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:15.398713 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:15.398548 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8fd0e2c-8495-4f2a-9d0e-166200ef8d59-kube-api-access-8wkwd" pod "c8fd0e2c-8495-4f2a-9d0e-166200ef8d59" (UID: "c8fd0e2c-8495-4f2a-9d0e-166200ef8d59"). InnerVolumeSpecName "kube-api-access-8wkwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:15.398713 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:15.398559 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8fd0e2c-8495-4f2a-9d0e-166200ef8d59-maas-api-xks-ca" pod "c8fd0e2c-8495-4f2a-9d0e-166200ef8d59" (UID: "c8fd0e2c-8495-4f2a-9d0e-166200ef8d59"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:45:15.498132 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:15.498095 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wkwd\" (UniqueName: \"kubernetes.io/projected/c8fd0e2c-8495-4f2a-9d0e-166200ef8d59-kube-api-access-8wkwd\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:15.498347 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:15.498328 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8fd0e2c-8495-4f2a-9d0e-166200ef8d59-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:15.498473 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:15.498457 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8fd0e2c-8495-4f2a-9d0e-166200ef8d59-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:15.541438 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:15.541402 5748 patch_prober.go:29] interesting pod/e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 21:45:15.541717 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:15.541689 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q" podUID="00245d73-3768-4ecd-9ddb-371c69b05e56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 21:45:15.796355 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:15.796323 5748 generic.go:356] "Generic (PLEG): container finished" podID="30fdce00-7aeb-4812-a580-8dde48515851" containerID="4c39af6db6e1904e5621fea658f06ff7ac071e3ec767c5d0ea3e85e3cfdae57c" exitCode=60 Oct 30 21:45:15.796696 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:15.796493 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d68156-29889945-2nc2b" event={"ID":"30fdce00-7aeb-4812-a580-8dde48515851","Type":"ContainerDied","Data":"4c39af6db6e1904e5621fea658f06ff7ac071e3ec767c5d0ea3e85e3cfdae57c"} Oct 30 21:45:15.796696 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:15.796538 5748 scope.go:122] "RemoveContainer" containerID="4e099e3d8725a46911addfe55675d67264679e0a59a62769239a715a00066676" Oct 30 21:45:15.796975 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:15.796960 5748 scope.go:122] "RemoveContainer" containerID="4c39af6db6e1904e5621fea658f06ff7ac071e3ec767c5d0ea3e85e3cfdae57c" Oct 30 21:45:15.797361 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:45:15.797336 5748 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-d68156-29889945-2nc2b_odh-ai-gateway-infra(30fdce00-7aeb-4812-a580-8dde48515851)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d68156-29889945-2nc2b" podUID="30fdce00-7aeb-4812-a580-8dde48515851" Oct 30 21:45:15.799888 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:15.799869 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60fb5a05-gdrnv" Oct 30 21:45:15.800004 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:15.799972 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60fb5a05-gdrnv" event={"ID":"c8fd0e2c-8495-4f2a-9d0e-166200ef8d59","Type":"ContainerDied","Data":"b2f65032d1fd12e15779f5097e380f845e6daf851026a6b568ea0c85661f4701"} Oct 30 21:45:15.800053 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:15.800010 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2f65032d1fd12e15779f5097e380f845e6daf851026a6b568ea0c85661f4701" Oct 30 21:45:15.895317 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:15.895282 5748 patch_prober.go:29] interesting pod/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 21:45:15.895473 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:15.895357 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9" podUID="0018c42f-45f5-4417-8b83-6d0cd9bd3c02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 21:45:16.171521 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:16.171491 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e843d27-d4nx9" Oct 30 21:45:16.359481 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:16.359377 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59ccc507-2625-4679-b3b8-b0796c1fd01a-kube-api-access-zz2ft\" (UniqueName: \"kubernetes.io/projected/59ccc507-2625-4679-b3b8-b0796c1fd01a-kube-api-access-zz2ft\") pod \"59ccc507-2625-4679-b3b8-b0796c1fd01a\" (UID: \"59ccc507-2625-4679-b3b8-b0796c1fd01a\") " Oct 30 21:45:16.363144 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:16.362617 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59ccc507-2625-4679-b3b8-b0796c1fd01a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59ccc507-2625-4679-b3b8-b0796c1fd01a-maas-api-xks-ca\") pod \"59ccc507-2625-4679-b3b8-b0796c1fd01a\" (UID: \"59ccc507-2625-4679-b3b8-b0796c1fd01a\") " Oct 30 21:45:16.363144 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:16.362722 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59ccc507-2625-4679-b3b8-b0796c1fd01a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59ccc507-2625-4679-b3b8-b0796c1fd01a-maas-api-openshift-ca\") pod \"59ccc507-2625-4679-b3b8-b0796c1fd01a\" (UID: \"59ccc507-2625-4679-b3b8-b0796c1fd01a\") " Oct 30 21:45:16.363610 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:16.363585 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59ccc507-2625-4679-b3b8-b0796c1fd01a-maas-api-openshift-ca" pod "59ccc507-2625-4679-b3b8-b0796c1fd01a" (UID: "59ccc507-2625-4679-b3b8-b0796c1fd01a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:16.365836 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:16.365793 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59ccc507-2625-4679-b3b8-b0796c1fd01a-kube-api-access-zz2ft" pod "59ccc507-2625-4679-b3b8-b0796c1fd01a" (UID: "59ccc507-2625-4679-b3b8-b0796c1fd01a"). InnerVolumeSpecName "kube-api-access-zz2ft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:16.366286 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:16.366266 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59ccc507-2625-4679-b3b8-b0796c1fd01a-maas-api-xks-ca" pod "59ccc507-2625-4679-b3b8-b0796c1fd01a" (UID: "59ccc507-2625-4679-b3b8-b0796c1fd01a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:45:16.367985 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:16.367962 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e86717c8-lxnvk"] Oct 30 21:45:16.370212 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:16.368720 5748 cpu_manager.go:414] "containerMap: removing container" podUID="c8fd0e2c-8495-4f2a-9d0e-166200ef8d59" containerName="revoke-keys" Oct 30 21:45:16.370212 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:16.368743 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8fd0e2c-8495-4f2a-9d0e-166200ef8d59" containerName="revoke-keys" Oct 30 21:45:16.370212 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:16.368762 5748 cpu_manager.go:414] "containerMap: removing container" podUID="59ccc507-2625-4679-b3b8-b0796c1fd01a" containerName="revoke-keys" Oct 30 21:45:16.370212 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:16.368769 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59ccc507-2625-4679-b3b8-b0796c1fd01a" containerName="revoke-keys" Oct 30 21:45:16.370212 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:16.369004 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8fd0e2c-8495-4f2a-9d0e-166200ef8d59" containerName="revoke-keys" Oct 30 21:45:16.370212 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:16.369021 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="59ccc507-2625-4679-b3b8-b0796c1fd01a" containerName="revoke-keys" Oct 30 21:45:16.373231 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:16.373213 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86717c8-lxnvk" Oct 30 21:45:16.465099 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:16.462611 5748 scope.go:122] "RemoveContainer" containerID="405baa727e42cbd0fc7d9cbfec6b47b4a3491465b31567cc76aec2eb76cd0ecc" Oct 30 21:45:16.465394 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:16.465311 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59ccc507-2625-4679-b3b8-b0796c1fd01a-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:16.465394 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:16.465340 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59ccc507-2625-4679-b3b8-b0796c1fd01a-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:16.465394 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:16.465356 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz2ft\" (UniqueName: \"kubernetes.io/projected/59ccc507-2625-4679-b3b8-b0796c1fd01a-kube-api-access-zz2ft\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:16.465556 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:16.465469 5748 scope.go:122] "RemoveContainer" containerID="1c68f6c08c762a73f7e4480dfbae471cb953fc6f28e91786da6bf43daf175e19" Oct 30 21:45:16.477250 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:16.477027 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e86717c8-lxnvk"] Oct 30 21:45:16.552920 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:16.549294 5748 patch_prober.go:29] interesting pod/e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 21:45:16.552920 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:16.549351 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q" podUID="00245d73-3768-4ecd-9ddb-371c69b05e56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 21:45:16.567575 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:16.567547 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5l74\" (UniqueName: \"kubernetes.io/projected/47875ea5-b355-4eef-af89-ff9dce4e91e7-kube-api-access-p5l74\") pod \"maas-api-revoke-sub-e86717c8-lxnvk\" (UID: \"47875ea5-b355-4eef-af89-ff9dce4e91e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86717c8-lxnvk" Oct 30 21:45:16.567831 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:16.567819 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47875ea5-b355-4eef-af89-ff9dce4e91e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e86717c8-lxnvk\" (UID: \"47875ea5-b355-4eef-af89-ff9dce4e91e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86717c8-lxnvk" Oct 30 21:45:16.568329 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:16.567995 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47875ea5-b355-4eef-af89-ff9dce4e91e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e86717c8-lxnvk\" (UID: \"47875ea5-b355-4eef-af89-ff9dce4e91e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86717c8-lxnvk" Oct 30 21:45:16.669783 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:16.669745 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5l74\" (UniqueName: \"kubernetes.io/projected/47875ea5-b355-4eef-af89-ff9dce4e91e7-kube-api-access-p5l74\") pod \"maas-api-revoke-sub-e86717c8-lxnvk\" (UID: \"47875ea5-b355-4eef-af89-ff9dce4e91e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86717c8-lxnvk" Oct 30 21:45:16.669961 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:16.669835 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47875ea5-b355-4eef-af89-ff9dce4e91e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e86717c8-lxnvk\" (UID: \"47875ea5-b355-4eef-af89-ff9dce4e91e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86717c8-lxnvk" Oct 30 21:45:16.669961 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:16.669910 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47875ea5-b355-4eef-af89-ff9dce4e91e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e86717c8-lxnvk\" (UID: \"47875ea5-b355-4eef-af89-ff9dce4e91e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86717c8-lxnvk" Oct 30 21:45:16.670885 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:16.670853 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47875ea5-b355-4eef-af89-ff9dce4e91e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e86717c8-lxnvk\" (UID: \"47875ea5-b355-4eef-af89-ff9dce4e91e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86717c8-lxnvk" Oct 30 21:45:16.704586 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:16.704498 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47875ea5-b355-4eef-af89-ff9dce4e91e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e86717c8-lxnvk\" (UID: \"47875ea5-b355-4eef-af89-ff9dce4e91e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86717c8-lxnvk" Oct 30 21:45:16.732088 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:16.732049 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5l74\" (UniqueName: \"kubernetes.io/projected/47875ea5-b355-4eef-af89-ff9dce4e91e7-kube-api-access-p5l74\") pod \"maas-api-revoke-sub-e86717c8-lxnvk\" (UID: \"47875ea5-b355-4eef-af89-ff9dce4e91e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86717c8-lxnvk" Oct 30 21:45:16.824762 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:16.824722 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e843d27-d4nx9" event={"ID":"59ccc507-2625-4679-b3b8-b0796c1fd01a","Type":"ContainerDied","Data":"e6615c5da5c21c55da025bb7e9a6ffe27397a06e0f61ddfeefdbab12bf846382"} Oct 30 21:45:16.825259 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:16.824819 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6615c5da5c21c55da025bb7e9a6ffe27397a06e0f61ddfeefdbab12bf846382" Oct 30 21:45:16.825259 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:16.824739 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e843d27-d4nx9" Oct 30 21:45:16.894838 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:16.894799 5748 patch_prober.go:29] interesting pod/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 21:45:16.895067 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:16.894869 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9" podUID="0018c42f-45f5-4417-8b83-6d0cd9bd3c02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 21:45:16.933004 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:16.932926 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d68156-29889945-2nc2b"] Oct 30 21:45:17.017411 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:17.008499 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86717c8-lxnvk" Oct 30 21:45:17.218816 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:17.218776 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d68156-29889945-2nc2b" Oct 30 21:45:17.284337 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:17.284305 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30fdce00-7aeb-4812-a580-8dde48515851-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30fdce00-7aeb-4812-a580-8dde48515851-maas-api-service-ca\") pod \"30fdce00-7aeb-4812-a580-8dde48515851\" (UID: \"30fdce00-7aeb-4812-a580-8dde48515851\") " Oct 30 21:45:17.284546 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:17.284422 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30fdce00-7aeb-4812-a580-8dde48515851-kube-api-access-596pn\" (UniqueName: \"kubernetes.io/projected/30fdce00-7aeb-4812-a580-8dde48515851-kube-api-access-596pn\") pod \"30fdce00-7aeb-4812-a580-8dde48515851\" (UID: \"30fdce00-7aeb-4812-a580-8dde48515851\") " Oct 30 21:45:17.284746 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:17.284722 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30fdce00-7aeb-4812-a580-8dde48515851-maas-api-service-ca" pod "30fdce00-7aeb-4812-a580-8dde48515851" (UID: "30fdce00-7aeb-4812-a580-8dde48515851"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:17.284966 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:17.284949 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30fdce00-7aeb-4812-a580-8dde48515851-maas-api-service-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:17.286814 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:17.286785 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30fdce00-7aeb-4812-a580-8dde48515851-kube-api-access-596pn" pod "30fdce00-7aeb-4812-a580-8dde48515851" (UID: "30fdce00-7aeb-4812-a580-8dde48515851"). InnerVolumeSpecName "kube-api-access-596pn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:17.386071 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:17.386037 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-596pn\" (UniqueName: \"kubernetes.io/projected/30fdce00-7aeb-4812-a580-8dde48515851-kube-api-access-596pn\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:17.434313 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:45:17.434249 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47875ea5_b355_4eef_af89_ff9dce4e91e7.slice/crio-05c70fe25d8343c1e36c669bb76bf1250528b4c222bc05ab65355d60990de524 WatchSource:0}: Error finding container 05c70fe25d8343c1e36c669bb76bf1250528b4c222bc05ab65355d60990de524: Status 404 returned error can't find the container with id 05c70fe25d8343c1e36c669bb76bf1250528b4c222bc05ab65355d60990de524 Oct 30 21:45:17.435368 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:17.435303 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e86717c8-lxnvk"] Oct 30 21:45:17.538096 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:17.538056 5748 patch_prober.go:29] interesting pod/e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 21:45:17.538268 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:17.538123 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q" podUID="00245d73-3768-4ecd-9ddb-371c69b05e56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 21:45:17.842285 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:17.842254 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d68156-29889945-2nc2b" Oct 30 21:45:17.842913 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:17.842266 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d68156-29889945-2nc2b" event={"ID":"30fdce00-7aeb-4812-a580-8dde48515851","Type":"ContainerDied","Data":"6b18a371cc0579afa521d73823d30cfc272f6719d4dc5b1f5a103b0d37e40643"} Oct 30 21:45:17.843059 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:17.842978 5748 scope.go:122] "RemoveContainer" containerID="4c39af6db6e1904e5621fea658f06ff7ac071e3ec767c5d0ea3e85e3cfdae57c" Oct 30 21:45:17.845825 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:17.845795 5748 generic.go:356] "Generic (PLEG): container finished" podID="dba2ae0f-a9ff-48da-b2e6-f8011473d615" containerID="cf369c18a3132538bc5e6f64e75d05dabb6735215556cdaff2ede1ded7700465" exitCode=60 Oct 30 21:45:17.846112 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:17.846080 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91874e-29889945-hbl8j" event={"ID":"dba2ae0f-a9ff-48da-b2e6-f8011473d615","Type":"ContainerDied","Data":"cf369c18a3132538bc5e6f64e75d05dabb6735215556cdaff2ede1ded7700465"} Oct 30 21:45:17.846592 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:17.846569 5748 scope.go:122] "RemoveContainer" containerID="cf369c18a3132538bc5e6f64e75d05dabb6735215556cdaff2ede1ded7700465" Oct 30 21:45:17.847014 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:45:17.846984 5748 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-91874e-29889945-hbl8j_odh-ai-gateway-infra(dba2ae0f-a9ff-48da-b2e6-f8011473d615)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91874e-29889945-hbl8j" podUID="dba2ae0f-a9ff-48da-b2e6-f8011473d615" Oct 30 21:45:17.850803 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:17.849794 5748 generic.go:356] "Generic (PLEG): container finished" podID="28a0c85c-d54f-42f9-8c57-1508fa1b2c82" containerID="dcddab3b01e0d4d8e3b3259c25acc48e1355934984d7856e8b06781850e43b41" exitCode=60 Oct 30 21:45:17.850803 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:17.849979 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889945-6lzds" event={"ID":"28a0c85c-d54f-42f9-8c57-1508fa1b2c82","Type":"ContainerDied","Data":"dcddab3b01e0d4d8e3b3259c25acc48e1355934984d7856e8b06781850e43b41"} Oct 30 21:45:17.850803 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:17.850398 5748 scope.go:122] "RemoveContainer" containerID="dcddab3b01e0d4d8e3b3259c25acc48e1355934984d7856e8b06781850e43b41" Oct 30 21:45:17.850803 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:45:17.850731 5748 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889945-6lzds_odh-ai-gateway-infra(28a0c85c-d54f-42f9-8c57-1508fa1b2c82)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889945-6lzds" podUID="28a0c85c-d54f-42f9-8c57-1508fa1b2c82" Oct 30 21:45:17.853341 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:17.853314 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86717c8-lxnvk" event={"ID":"47875ea5-b355-4eef-af89-ff9dce4e91e7","Type":"ContainerStarted","Data":"13da6e3babb1c15b3b00be48dd3606c2d7619469994ba1289391a5ecff6023a0"} Oct 30 21:45:17.853447 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:17.853366 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86717c8-lxnvk" event={"ID":"47875ea5-b355-4eef-af89-ff9dce4e91e7","Type":"ContainerStarted","Data":"05c70fe25d8343c1e36c669bb76bf1250528b4c222bc05ab65355d60990de524"} Oct 30 21:45:17.860677 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:17.860655 5748 scope.go:122] "RemoveContainer" containerID="405baa727e42cbd0fc7d9cbfec6b47b4a3491465b31567cc76aec2eb76cd0ecc" Oct 30 21:45:17.882429 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:17.882402 5748 scope.go:122] "RemoveContainer" containerID="1c68f6c08c762a73f7e4480dfbae471cb953fc6f28e91786da6bf43daf175e19" Oct 30 21:45:18.141874 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:18.141822 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86717c8-lxnvk" podStartSLOduration=2.141807906 podStartE2EDuration="2.141807906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:18.139855289 +0000 UTC m=+1301.882376223" watchObservedRunningTime="2026-10-30 21:45:18.141807906 +0000 UTC m=+1301.884328841" Oct 30 21:45:18.170953 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:18.170226 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9" Oct 30 21:45:18.171517 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:18.171494 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9" Oct 30 21:45:18.172875 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:18.172851 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9" Oct 30 21:45:18.256760 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:18.256726 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d68156-29889945-2nc2b"] Oct 30 21:45:18.263917 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:18.263880 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d68156-29889945-2nc2b"] Oct 30 21:45:18.422007 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:18.421869 5748 scope.go:122] "RemoveContainer" containerID="e5bf67f97340cd037ff2871e8d7bc729214a2e1494fb3f544c4f1787f016bcb9" Oct 30 21:45:18.428994 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:18.428972 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30fdce00-7aeb-4812-a580-8dde48515851" path="/var/lib/kubelet/pods/30fdce00-7aeb-4812-a580-8dde48515851/volumes" Oct 30 21:45:18.546405 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:18.546363 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q" Oct 30 21:45:18.547395 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:18.547369 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q" Oct 30 21:45:18.550126 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:18.550094 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q" Oct 30 21:45:18.866152 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:18.866108 5748 generic.go:356] "Generic (PLEG): container finished" podID="47875ea5-b355-4eef-af89-ff9dce4e91e7" containerID="13da6e3babb1c15b3b00be48dd3606c2d7619469994ba1289391a5ecff6023a0" exitCode=0 Oct 30 21:45:18.866576 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:18.866174 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86717c8-lxnvk" event={"ID":"47875ea5-b355-4eef-af89-ff9dce4e91e7","Type":"ContainerDied","Data":"13da6e3babb1c15b3b00be48dd3606c2d7619469994ba1289391a5ecff6023a0"} Oct 30 21:45:18.868669 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:18.868643 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-691598-29889945-vzhl4" event={"ID":"c042e96b-236e-419e-b531-43798f9bb25f","Type":"ContainerStarted","Data":"88fde1b95ce31340959a979f1f4aa601135ef1efd4e0029a047f9f5f8aa25d14"} Oct 30 21:45:19.033376 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:19.033301 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889945-6lzds"] Oct 30 21:45:19.195089 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:19.195059 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889945-6lzds" Oct 30 21:45:19.216186 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:19.216141 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28a0c85c-d54f-42f9-8c57-1508fa1b2c82-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28a0c85c-d54f-42f9-8c57-1508fa1b2c82-maas-api-service-ca\") pod \"28a0c85c-d54f-42f9-8c57-1508fa1b2c82\" (UID: \"28a0c85c-d54f-42f9-8c57-1508fa1b2c82\") " Oct 30 21:45:19.216351 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:19.216206 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28a0c85c-d54f-42f9-8c57-1508fa1b2c82-kube-api-access-f6gnp\" (UniqueName: \"kubernetes.io/projected/28a0c85c-d54f-42f9-8c57-1508fa1b2c82-kube-api-access-f6gnp\") pod \"28a0c85c-d54f-42f9-8c57-1508fa1b2c82\" (UID: \"28a0c85c-d54f-42f9-8c57-1508fa1b2c82\") " Oct 30 21:45:19.216491 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:19.216469 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28a0c85c-d54f-42f9-8c57-1508fa1b2c82-maas-api-service-ca" pod "28a0c85c-d54f-42f9-8c57-1508fa1b2c82" (UID: "28a0c85c-d54f-42f9-8c57-1508fa1b2c82"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:19.216846 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:19.216827 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28a0c85c-d54f-42f9-8c57-1508fa1b2c82-maas-api-service-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:19.218949 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:19.218930 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28a0c85c-d54f-42f9-8c57-1508fa1b2c82-kube-api-access-f6gnp" pod "28a0c85c-d54f-42f9-8c57-1508fa1b2c82" (UID: "28a0c85c-d54f-42f9-8c57-1508fa1b2c82"). InnerVolumeSpecName "kube-api-access-f6gnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:19.318477 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:19.318450 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6gnp\" (UniqueName: \"kubernetes.io/projected/28a0c85c-d54f-42f9-8c57-1508fa1b2c82-kube-api-access-f6gnp\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:19.390400 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:19.390364 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91874e-29889945-hbl8j"] Oct 30 21:45:19.560337 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:19.560267 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91874e-29889945-hbl8j" Oct 30 21:45:19.622137 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:19.622097 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dba2ae0f-a9ff-48da-b2e6-f8011473d615-kube-api-access-bpkfz\" (UniqueName: \"kubernetes.io/projected/dba2ae0f-a9ff-48da-b2e6-f8011473d615-kube-api-access-bpkfz\") pod \"dba2ae0f-a9ff-48da-b2e6-f8011473d615\" (UID: \"dba2ae0f-a9ff-48da-b2e6-f8011473d615\") " Oct 30 21:45:19.622287 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:19.622242 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dba2ae0f-a9ff-48da-b2e6-f8011473d615-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dba2ae0f-a9ff-48da-b2e6-f8011473d615-maas-api-service-ca\") pod \"dba2ae0f-a9ff-48da-b2e6-f8011473d615\" (UID: \"dba2ae0f-a9ff-48da-b2e6-f8011473d615\") " Oct 30 21:45:19.622601 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:19.622574 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dba2ae0f-a9ff-48da-b2e6-f8011473d615-maas-api-service-ca" pod "dba2ae0f-a9ff-48da-b2e6-f8011473d615" (UID: "dba2ae0f-a9ff-48da-b2e6-f8011473d615"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:19.622884 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:19.622866 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dba2ae0f-a9ff-48da-b2e6-f8011473d615-maas-api-service-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:19.624262 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:19.624241 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dba2ae0f-a9ff-48da-b2e6-f8011473d615-kube-api-access-bpkfz" pod "dba2ae0f-a9ff-48da-b2e6-f8011473d615" (UID: "dba2ae0f-a9ff-48da-b2e6-f8011473d615"). InnerVolumeSpecName "kube-api-access-bpkfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:19.723618 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:19.723584 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpkfz\" (UniqueName: \"kubernetes.io/projected/dba2ae0f-a9ff-48da-b2e6-f8011473d615-kube-api-access-bpkfz\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:19.882145 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:19.882055 5748 generic.go:356] "Generic (PLEG): container finished" podID="c042e96b-236e-419e-b531-43798f9bb25f" containerID="88fde1b95ce31340959a979f1f4aa601135ef1efd4e0029a047f9f5f8aa25d14" exitCode=60 Oct 30 21:45:19.882145 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:19.882128 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-691598-29889945-vzhl4" event={"ID":"c042e96b-236e-419e-b531-43798f9bb25f","Type":"ContainerDied","Data":"88fde1b95ce31340959a979f1f4aa601135ef1efd4e0029a047f9f5f8aa25d14"} Oct 30 21:45:19.882728 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:19.882193 5748 scope.go:122] "RemoveContainer" containerID="e5bf67f97340cd037ff2871e8d7bc729214a2e1494fb3f544c4f1787f016bcb9" Oct 30 21:45:19.882728 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:19.882611 5748 scope.go:122] "RemoveContainer" containerID="88fde1b95ce31340959a979f1f4aa601135ef1efd4e0029a047f9f5f8aa25d14" Oct 30 21:45:19.883063 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:45:19.883036 5748 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-691598-29889945-vzhl4_odh-ai-gateway-infra(c042e96b-236e-419e-b531-43798f9bb25f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-691598-29889945-vzhl4" podUID="c042e96b-236e-419e-b531-43798f9bb25f" Oct 30 21:45:19.884557 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:19.884537 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91874e-29889945-hbl8j" Oct 30 21:45:19.884655 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:19.884566 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91874e-29889945-hbl8j" event={"ID":"dba2ae0f-a9ff-48da-b2e6-f8011473d615","Type":"ContainerDied","Data":"4b47561863418238acca62312e1a17d4942c1beab182479336b1ba90e0477fb9"} Oct 30 21:45:19.886274 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:19.886251 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889945-6lzds" Oct 30 21:45:19.886364 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:19.886291 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889945-6lzds" event={"ID":"28a0c85c-d54f-42f9-8c57-1508fa1b2c82","Type":"ContainerDied","Data":"e52d9c1d65ab42163336a142295af1a0ac6a72c36ba4016bf753a79f63b74881"} Oct 30 21:45:19.900732 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:19.900704 5748 scope.go:122] "RemoveContainer" containerID="cf369c18a3132538bc5e6f64e75d05dabb6735215556cdaff2ede1ded7700465" Oct 30 21:45:19.913545 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:19.913517 5748 scope.go:122] "RemoveContainer" containerID="dcddab3b01e0d4d8e3b3259c25acc48e1355934984d7856e8b06781850e43b41" Oct 30 21:45:20.023493 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:20.023467 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86717c8-lxnvk" Oct 30 21:45:20.056773 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:20.056744 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6be7c7d8-q65x9"] Oct 30 21:45:20.061912 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:20.061872 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6be7c7d8-q65x9"] Oct 30 21:45:20.119535 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:20.119497 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889945-6lzds"] Oct 30 21:45:20.126765 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:20.126734 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889945-6lzds"] Oct 30 21:45:20.130466 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:20.130445 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47875ea5-b355-4eef-af89-ff9dce4e91e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47875ea5-b355-4eef-af89-ff9dce4e91e7-maas-api-openshift-ca\") pod \"47875ea5-b355-4eef-af89-ff9dce4e91e7\" (UID: \"47875ea5-b355-4eef-af89-ff9dce4e91e7\") " Oct 30 21:45:20.130616 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:20.130599 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47875ea5-b355-4eef-af89-ff9dce4e91e7-kube-api-access-p5l74\" (UniqueName: \"kubernetes.io/projected/47875ea5-b355-4eef-af89-ff9dce4e91e7-kube-api-access-p5l74\") pod \"47875ea5-b355-4eef-af89-ff9dce4e91e7\" (UID: \"47875ea5-b355-4eef-af89-ff9dce4e91e7\") " Oct 30 21:45:20.130672 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:20.130661 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47875ea5-b355-4eef-af89-ff9dce4e91e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47875ea5-b355-4eef-af89-ff9dce4e91e7-maas-api-xks-ca\") pod \"47875ea5-b355-4eef-af89-ff9dce4e91e7\" (UID: \"47875ea5-b355-4eef-af89-ff9dce4e91e7\") " Oct 30 21:45:20.130833 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:20.130808 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47875ea5-b355-4eef-af89-ff9dce4e91e7-maas-api-openshift-ca" pod "47875ea5-b355-4eef-af89-ff9dce4e91e7" (UID: "47875ea5-b355-4eef-af89-ff9dce4e91e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:20.131145 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:20.131129 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47875ea5-b355-4eef-af89-ff9dce4e91e7-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:20.132726 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:20.132679 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47875ea5-b355-4eef-af89-ff9dce4e91e7-maas-api-xks-ca" pod "47875ea5-b355-4eef-af89-ff9dce4e91e7" (UID: "47875ea5-b355-4eef-af89-ff9dce4e91e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:45:20.133075 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:20.133057 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47875ea5-b355-4eef-af89-ff9dce4e91e7-kube-api-access-p5l74" pod "47875ea5-b355-4eef-af89-ff9dce4e91e7" (UID: "47875ea5-b355-4eef-af89-ff9dce4e91e7"). InnerVolumeSpecName "kube-api-access-p5l74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:20.145388 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:20.145364 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-691598-29889945-vzhl4"] Oct 30 21:45:20.232708 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:20.232651 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47875ea5-b355-4eef-af89-ff9dce4e91e7-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:20.232922 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:20.232687 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5l74\" (UniqueName: \"kubernetes.io/projected/47875ea5-b355-4eef-af89-ff9dce4e91e7-kube-api-access-p5l74\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:20.237369 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:20.237342 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91874e-29889945-hbl8j"] Oct 30 21:45:20.242402 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:20.242379 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91874e-29889945-hbl8j"] Oct 30 21:45:20.426776 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:20.426742 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28a0c85c-d54f-42f9-8c57-1508fa1b2c82" path="/var/lib/kubelet/pods/28a0c85c-d54f-42f9-8c57-1508fa1b2c82/volumes" Oct 30 21:45:20.427287 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:20.427271 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d05672e8-e4f1-465c-acca-1c7e0c4858bf" path="/var/lib/kubelet/pods/d05672e8-e4f1-465c-acca-1c7e0c4858bf/volumes" Oct 30 21:45:20.427633 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:20.427621 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dba2ae0f-a9ff-48da-b2e6-f8011473d615" path="/var/lib/kubelet/pods/dba2ae0f-a9ff-48da-b2e6-f8011473d615/volumes" Oct 30 21:45:20.899644 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:20.899611 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86717c8-lxnvk" event={"ID":"47875ea5-b355-4eef-af89-ff9dce4e91e7","Type":"ContainerDied","Data":"05c70fe25d8343c1e36c669bb76bf1250528b4c222bc05ab65355d60990de524"} Oct 30 21:45:20.900070 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:20.899662 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86717c8-lxnvk" Oct 30 21:45:20.900070 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:20.899666 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05c70fe25d8343c1e36c669bb76bf1250528b4c222bc05ab65355d60990de524" Oct 30 21:45:21.069727 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.069689 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-691598-29889945-vzhl4" Oct 30 21:45:21.143967 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.143936 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c042e96b-236e-419e-b531-43798f9bb25f-kube-api-access-sx9w8\" (UniqueName: \"kubernetes.io/projected/c042e96b-236e-419e-b531-43798f9bb25f-kube-api-access-sx9w8\") pod \"c042e96b-236e-419e-b531-43798f9bb25f\" (UID: \"c042e96b-236e-419e-b531-43798f9bb25f\") " Oct 30 21:45:21.144117 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.144085 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c042e96b-236e-419e-b531-43798f9bb25f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c042e96b-236e-419e-b531-43798f9bb25f-maas-api-service-ca\") pod \"c042e96b-236e-419e-b531-43798f9bb25f\" (UID: \"c042e96b-236e-419e-b531-43798f9bb25f\") " Oct 30 21:45:21.144457 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.144427 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c042e96b-236e-419e-b531-43798f9bb25f-maas-api-service-ca" pod "c042e96b-236e-419e-b531-43798f9bb25f" (UID: "c042e96b-236e-419e-b531-43798f9bb25f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:21.146113 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.146093 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c042e96b-236e-419e-b531-43798f9bb25f-kube-api-access-sx9w8" pod "c042e96b-236e-419e-b531-43798f9bb25f" (UID: "c042e96b-236e-419e-b531-43798f9bb25f"). InnerVolumeSpecName "kube-api-access-sx9w8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:21.245859 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.245821 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c042e96b-236e-419e-b531-43798f9bb25f-maas-api-service-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:21.245859 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.245858 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx9w8\" (UniqueName: \"kubernetes.io/projected/c042e96b-236e-419e-b531-43798f9bb25f-kube-api-access-sx9w8\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:21.536797 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.536717 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e7250d4-jggkz"] Oct 30 21:45:21.537303 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.537277 5748 cpu_manager.go:414] "containerMap: removing container" podUID="28a0c85c-d54f-42f9-8c57-1508fa1b2c82" containerName="cleanup" Oct 30 21:45:21.537303 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.537305 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28a0c85c-d54f-42f9-8c57-1508fa1b2c82" containerName="cleanup" Oct 30 21:45:21.537399 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.537320 5748 cpu_manager.go:414] "containerMap: removing container" podUID="dba2ae0f-a9ff-48da-b2e6-f8011473d615" containerName="cleanup" Oct 30 21:45:21.537399 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.537326 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dba2ae0f-a9ff-48da-b2e6-f8011473d615" containerName="cleanup" Oct 30 21:45:21.537399 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.537342 5748 cpu_manager.go:414] "containerMap: removing container" podUID="dba2ae0f-a9ff-48da-b2e6-f8011473d615" containerName="cleanup" Oct 30 21:45:21.537399 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.537346 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dba2ae0f-a9ff-48da-b2e6-f8011473d615" containerName="cleanup" Oct 30 21:45:21.537399 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.537355 5748 cpu_manager.go:414] "containerMap: removing container" podUID="30fdce00-7aeb-4812-a580-8dde48515851" containerName="cleanup" Oct 30 21:45:21.537399 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.537359 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30fdce00-7aeb-4812-a580-8dde48515851" containerName="cleanup" Oct 30 21:45:21.537399 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.537369 5748 cpu_manager.go:414] "containerMap: removing container" podUID="28a0c85c-d54f-42f9-8c57-1508fa1b2c82" containerName="cleanup" Oct 30 21:45:21.537399 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.537373 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28a0c85c-d54f-42f9-8c57-1508fa1b2c82" containerName="cleanup" Oct 30 21:45:21.537399 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.537381 5748 cpu_manager.go:414] "containerMap: removing container" podUID="c042e96b-236e-419e-b531-43798f9bb25f" containerName="cleanup" Oct 30 21:45:21.537399 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.537386 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c042e96b-236e-419e-b531-43798f9bb25f" containerName="cleanup" Oct 30 21:45:21.537399 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.537392 5748 cpu_manager.go:414] "containerMap: removing container" podUID="30fdce00-7aeb-4812-a580-8dde48515851" containerName="cleanup" Oct 30 21:45:21.537399 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.537396 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30fdce00-7aeb-4812-a580-8dde48515851" containerName="cleanup" Oct 30 21:45:21.537743 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.537408 5748 cpu_manager.go:414] "containerMap: removing container" podUID="30fdce00-7aeb-4812-a580-8dde48515851" containerName="cleanup" Oct 30 21:45:21.537743 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.537412 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30fdce00-7aeb-4812-a580-8dde48515851" containerName="cleanup" Oct 30 21:45:21.537743 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.537426 5748 cpu_manager.go:414] "containerMap: removing container" podUID="47875ea5-b355-4eef-af89-ff9dce4e91e7" containerName="revoke-keys" Oct 30 21:45:21.537743 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.537431 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47875ea5-b355-4eef-af89-ff9dce4e91e7" containerName="revoke-keys" Oct 30 21:45:21.537743 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.537437 5748 cpu_manager.go:414] "containerMap: removing container" podUID="c042e96b-236e-419e-b531-43798f9bb25f" containerName="cleanup" Oct 30 21:45:21.537743 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.537441 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c042e96b-236e-419e-b531-43798f9bb25f" containerName="cleanup" Oct 30 21:45:21.537743 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.537449 5748 cpu_manager.go:414] "containerMap: removing container" podUID="c042e96b-236e-419e-b531-43798f9bb25f" containerName="cleanup" Oct 30 21:45:21.537743 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.537453 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c042e96b-236e-419e-b531-43798f9bb25f" containerName="cleanup" Oct 30 21:45:21.537743 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.537529 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="28a0c85c-d54f-42f9-8c57-1508fa1b2c82" containerName="cleanup" Oct 30 21:45:21.537743 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.537538 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="dba2ae0f-a9ff-48da-b2e6-f8011473d615" containerName="cleanup" Oct 30 21:45:21.537743 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.537545 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="dba2ae0f-a9ff-48da-b2e6-f8011473d615" containerName="cleanup" Oct 30 21:45:21.537743 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.537551 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="30fdce00-7aeb-4812-a580-8dde48515851" containerName="cleanup" Oct 30 21:45:21.537743 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.537556 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="c042e96b-236e-419e-b531-43798f9bb25f" containerName="cleanup" Oct 30 21:45:21.537743 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.537562 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="28a0c85c-d54f-42f9-8c57-1508fa1b2c82" containerName="cleanup" Oct 30 21:45:21.537743 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.537567 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="30fdce00-7aeb-4812-a580-8dde48515851" containerName="cleanup" Oct 30 21:45:21.537743 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.537574 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="47875ea5-b355-4eef-af89-ff9dce4e91e7" containerName="revoke-keys" Oct 30 21:45:21.537743 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.537581 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="c042e96b-236e-419e-b531-43798f9bb25f" containerName="cleanup" Oct 30 21:45:21.537743 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.537586 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="c042e96b-236e-419e-b531-43798f9bb25f" containerName="cleanup" Oct 30 21:45:21.541123 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.541104 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e7250d4-jggkz" Oct 30 21:45:21.596084 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.596048 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:45:21.599820 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.599795 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e7250d4-jggkz"] Oct 30 21:45:21.650933 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.650877 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/097d56ed-7515-4b94-9176-56e1c639d24c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e7250d4-jggkz\" (UID: \"097d56ed-7515-4b94-9176-56e1c639d24c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e7250d4-jggkz" Oct 30 21:45:21.651115 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.651077 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/097d56ed-7515-4b94-9176-56e1c639d24c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e7250d4-jggkz\" (UID: \"097d56ed-7515-4b94-9176-56e1c639d24c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e7250d4-jggkz" Oct 30 21:45:21.651260 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.651242 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrmzz\" (UniqueName: \"kubernetes.io/projected/097d56ed-7515-4b94-9176-56e1c639d24c-kube-api-access-zrmzz\") pod \"maas-api-revoke-sub-6e7250d4-jggkz\" (UID: \"097d56ed-7515-4b94-9176-56e1c639d24c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e7250d4-jggkz" Oct 30 21:45:21.752880 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.752849 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrmzz\" (UniqueName: \"kubernetes.io/projected/097d56ed-7515-4b94-9176-56e1c639d24c-kube-api-access-zrmzz\") pod \"maas-api-revoke-sub-6e7250d4-jggkz\" (UID: \"097d56ed-7515-4b94-9176-56e1c639d24c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e7250d4-jggkz" Oct 30 21:45:21.753083 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.753049 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/097d56ed-7515-4b94-9176-56e1c639d24c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e7250d4-jggkz\" (UID: \"097d56ed-7515-4b94-9176-56e1c639d24c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e7250d4-jggkz" Oct 30 21:45:21.753183 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.753162 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/097d56ed-7515-4b94-9176-56e1c639d24c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e7250d4-jggkz\" (UID: \"097d56ed-7515-4b94-9176-56e1c639d24c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e7250d4-jggkz" Oct 30 21:45:21.753883 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.753854 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/097d56ed-7515-4b94-9176-56e1c639d24c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e7250d4-jggkz\" (UID: \"097d56ed-7515-4b94-9176-56e1c639d24c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e7250d4-jggkz" Oct 30 21:45:21.755535 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.755515 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/097d56ed-7515-4b94-9176-56e1c639d24c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e7250d4-jggkz\" (UID: \"097d56ed-7515-4b94-9176-56e1c639d24c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e7250d4-jggkz" Oct 30 21:45:21.815192 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.815107 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrmzz\" (UniqueName: \"kubernetes.io/projected/097d56ed-7515-4b94-9176-56e1c639d24c-kube-api-access-zrmzz\") pod \"maas-api-revoke-sub-6e7250d4-jggkz\" (UID: \"097d56ed-7515-4b94-9176-56e1c639d24c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e7250d4-jggkz" Oct 30 21:45:21.854469 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.854418 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e7250d4-jggkz" Oct 30 21:45:21.917206 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.917164 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-691598-29889945-vzhl4" event={"ID":"c042e96b-236e-419e-b531-43798f9bb25f","Type":"ContainerDied","Data":"de51a9b27b4ce608b8c7e17908edceccc01c0fae60f37b36450387adb06364ff"} Oct 30 21:45:21.917753 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.917258 5748 scope.go:122] "RemoveContainer" containerID="88fde1b95ce31340959a979f1f4aa601135ef1efd4e0029a047f9f5f8aa25d14" Oct 30 21:45:21.917753 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.917421 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-691598-29889945-vzhl4" Oct 30 21:45:21.924882 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.923356 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a488c3f-54tr8"] Oct 30 21:45:21.924882 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.924249 5748 cpu_manager.go:414] "containerMap: removing container" podUID="dba2ae0f-a9ff-48da-b2e6-f8011473d615" containerName="cleanup" Oct 30 21:45:21.924882 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.924266 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dba2ae0f-a9ff-48da-b2e6-f8011473d615" containerName="cleanup" Oct 30 21:45:21.924882 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.924286 5748 cpu_manager.go:414] "containerMap: removing container" podUID="28a0c85c-d54f-42f9-8c57-1508fa1b2c82" containerName="cleanup" Oct 30 21:45:21.924882 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.924294 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28a0c85c-d54f-42f9-8c57-1508fa1b2c82" containerName="cleanup" Oct 30 21:45:21.924882 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.924493 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="dba2ae0f-a9ff-48da-b2e6-f8011473d615" containerName="cleanup" Oct 30 21:45:21.924882 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.924507 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="30fdce00-7aeb-4812-a580-8dde48515851" containerName="cleanup" Oct 30 21:45:21.924882 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.924519 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="28a0c85c-d54f-42f9-8c57-1508fa1b2c82" containerName="cleanup" Oct 30 21:45:21.946509 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.946481 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a488c3f-54tr8" Oct 30 21:45:21.980558 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:21.980516 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a488c3f-54tr8"] Oct 30 21:45:22.057451 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:22.057407 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3880a175-d299-45eb-a49a-a89b079b67e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a488c3f-54tr8\" (UID: \"3880a175-d299-45eb-a49a-a89b079b67e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a488c3f-54tr8" Oct 30 21:45:22.057643 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:22.057575 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3880a175-d299-45eb-a49a-a89b079b67e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a488c3f-54tr8\" (UID: \"3880a175-d299-45eb-a49a-a89b079b67e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a488c3f-54tr8" Oct 30 21:45:22.057742 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:22.057727 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nctks\" (UniqueName: \"kubernetes.io/projected/3880a175-d299-45eb-a49a-a89b079b67e7-kube-api-access-nctks\") pod \"maas-api-revoke-sub-6a488c3f-54tr8\" (UID: \"3880a175-d299-45eb-a49a-a89b079b67e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a488c3f-54tr8" Oct 30 21:45:22.093107 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:22.093020 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-691598-29889945-vzhl4"] Oct 30 21:45:22.100589 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:22.100557 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-691598-29889945-vzhl4"] Oct 30 21:45:22.158923 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:22.158876 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nctks\" (UniqueName: \"kubernetes.io/projected/3880a175-d299-45eb-a49a-a89b079b67e7-kube-api-access-nctks\") pod \"maas-api-revoke-sub-6a488c3f-54tr8\" (UID: \"3880a175-d299-45eb-a49a-a89b079b67e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a488c3f-54tr8" Oct 30 21:45:22.159104 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:22.159005 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3880a175-d299-45eb-a49a-a89b079b67e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a488c3f-54tr8\" (UID: \"3880a175-d299-45eb-a49a-a89b079b67e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a488c3f-54tr8" Oct 30 21:45:22.159169 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:22.159111 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3880a175-d299-45eb-a49a-a89b079b67e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a488c3f-54tr8\" (UID: \"3880a175-d299-45eb-a49a-a89b079b67e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a488c3f-54tr8" Oct 30 21:45:22.159802 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:22.159775 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3880a175-d299-45eb-a49a-a89b079b67e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a488c3f-54tr8\" (UID: \"3880a175-d299-45eb-a49a-a89b079b67e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a488c3f-54tr8" Oct 30 21:45:22.161630 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:22.161608 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3880a175-d299-45eb-a49a-a89b079b67e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a488c3f-54tr8\" (UID: \"3880a175-d299-45eb-a49a-a89b079b67e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a488c3f-54tr8" Oct 30 21:45:22.179481 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:22.179443 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx"] Oct 30 21:45:22.183839 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:22.183817 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx" Oct 30 21:45:22.221115 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:22.221085 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nctks\" (UniqueName: \"kubernetes.io/projected/3880a175-d299-45eb-a49a-a89b079b67e7-kube-api-access-nctks\") pod \"maas-api-revoke-sub-6a488c3f-54tr8\" (UID: \"3880a175-d299-45eb-a49a-a89b079b67e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a488c3f-54tr8" Oct 30 21:45:22.240514 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:22.240459 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-6ea374-openshift-default-dockercfg-jjxj8\"" Oct 30 21:45:22.246481 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:22.246454 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx"] Oct 30 21:45:22.246585 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:45:22.246510 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod097d56ed_7515_4b94_9176_56e1c639d24c.slice/crio-03b6bb7a2789888e7bca443cded06603985076bce30f960eabe64d9c4f153ece WatchSource:0}: Error finding container 03b6bb7a2789888e7bca443cded06603985076bce30f960eabe64d9c4f153ece: Status 404 returned error can't find the container with id 03b6bb7a2789888e7bca443cded06603985076bce30f960eabe64d9c4f153ece Oct 30 21:45:22.250729 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:22.250703 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e7250d4-jggkz"] Oct 30 21:45:22.260223 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:22.260192 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-istio-podinfo\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx\" (UID: \"6960e4e7-1cd0-48b0-85f8-c88e46102ad4\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx" Oct 30 21:45:22.260347 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:22.260247 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-istio-envoy\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx\" (UID: \"6960e4e7-1cd0-48b0-85f8-c88e46102ad4\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx" Oct 30 21:45:22.260445 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:22.260420 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-istio-data\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx\" (UID: \"6960e4e7-1cd0-48b0-85f8-c88e46102ad4\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx" Oct 30 21:45:22.260503 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:22.260427 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a488c3f-54tr8" Oct 30 21:45:22.260602 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:22.260475 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cp8h\" (UniqueName: \"kubernetes.io/projected/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-kube-api-access-7cp8h\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx\" (UID: \"6960e4e7-1cd0-48b0-85f8-c88e46102ad4\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx" Oct 30 21:45:22.260725 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:22.260639 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-workload-socket\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx\" (UID: \"6960e4e7-1cd0-48b0-85f8-c88e46102ad4\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx" Oct 30 21:45:22.260842 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:22.260721 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-credential-socket\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx\" (UID: \"6960e4e7-1cd0-48b0-85f8-c88e46102ad4\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx" Oct 30 21:45:22.260842 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:22.260765 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-workload-certs\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx\" (UID: \"6960e4e7-1cd0-48b0-85f8-c88e46102ad4\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx" Oct 30 21:45:22.261067 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:22.260999 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-istio-token\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx\" (UID: \"6960e4e7-1cd0-48b0-85f8-c88e46102ad4\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx" Oct 30 21:45:22.261161 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:22.261077 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-istiod-ca-cert\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx\" (UID: \"6960e4e7-1cd0-48b0-85f8-c88e46102ad4\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx" Oct 30 21:45:22.363416 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:22.362546 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-istio-token\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx\" (UID: \"6960e4e7-1cd0-48b0-85f8-c88e46102ad4\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx" Oct 30 21:45:22.363416 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:22.362613 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-istiod-ca-cert\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx\" (UID: \"6960e4e7-1cd0-48b0-85f8-c88e46102ad4\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx" Oct 30 21:45:22.363416 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:22.362661 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-istio-podinfo\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx\" (UID: \"6960e4e7-1cd0-48b0-85f8-c88e46102ad4\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx" Oct 30 21:45:22.363416 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:22.362690 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-istio-envoy\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx\" (UID: \"6960e4e7-1cd0-48b0-85f8-c88e46102ad4\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx" Oct 30 21:45:22.363416 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:22.362727 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-istio-data\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx\" (UID: \"6960e4e7-1cd0-48b0-85f8-c88e46102ad4\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx" Oct 30 21:45:22.363416 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:22.362752 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cp8h\" (UniqueName: \"kubernetes.io/projected/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-kube-api-access-7cp8h\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx\" (UID: \"6960e4e7-1cd0-48b0-85f8-c88e46102ad4\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx" Oct 30 21:45:22.363416 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:22.362772 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-workload-socket\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx\" (UID: \"6960e4e7-1cd0-48b0-85f8-c88e46102ad4\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx" Oct 30 21:45:22.363416 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:22.362818 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-credential-socket\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx\" (UID: \"6960e4e7-1cd0-48b0-85f8-c88e46102ad4\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx" Oct 30 21:45:22.363416 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:22.362844 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-workload-certs\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx\" (UID: \"6960e4e7-1cd0-48b0-85f8-c88e46102ad4\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx" Oct 30 21:45:22.363416 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:22.363357 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-istio-data\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx\" (UID: \"6960e4e7-1cd0-48b0-85f8-c88e46102ad4\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx" Oct 30 21:45:22.364063 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:22.363495 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-istiod-ca-cert\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx\" (UID: \"6960e4e7-1cd0-48b0-85f8-c88e46102ad4\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx" Oct 30 21:45:22.364063 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:22.363519 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-workload-certs\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx\" (UID: \"6960e4e7-1cd0-48b0-85f8-c88e46102ad4\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx" Oct 30 21:45:22.364063 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:22.363722 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-workload-socket\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx\" (UID: \"6960e4e7-1cd0-48b0-85f8-c88e46102ad4\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx" Oct 30 21:45:22.364063 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:22.363814 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-credential-socket\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx\" (UID: \"6960e4e7-1cd0-48b0-85f8-c88e46102ad4\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx" Oct 30 21:45:22.365627 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:22.365608 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-istio-envoy\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx\" (UID: \"6960e4e7-1cd0-48b0-85f8-c88e46102ad4\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx" Oct 30 21:45:22.366163 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:22.366127 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-istio-podinfo\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx\" (UID: \"6960e4e7-1cd0-48b0-85f8-c88e46102ad4\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx" Oct 30 21:45:22.421449 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:22.421408 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c042e96b-236e-419e-b531-43798f9bb25f" path="/var/lib/kubelet/pods/c042e96b-236e-419e-b531-43798f9bb25f/volumes" Oct 30 21:45:22.424554 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:22.424523 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-istio-token\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx\" (UID: \"6960e4e7-1cd0-48b0-85f8-c88e46102ad4\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx" Oct 30 21:45:22.424883 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:22.424843 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cp8h\" (UniqueName: \"kubernetes.io/projected/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-kube-api-access-7cp8h\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx\" (UID: \"6960e4e7-1cd0-48b0-85f8-c88e46102ad4\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx" Oct 30 21:45:22.499571 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:22.499534 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx" Oct 30 21:45:22.652507 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:22.652475 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a488c3f-54tr8"] Oct 30 21:45:22.652928 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:45:22.652870 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3880a175_d299_45eb_a49a_a89b079b67e7.slice/crio-098db84ceb7ccd01f174a135547c32bea9268379ef33f8125eae251a519dac8f WatchSource:0}: Error finding container 098db84ceb7ccd01f174a135547c32bea9268379ef33f8125eae251a519dac8f: Status 404 returned error can't find the container with id 098db84ceb7ccd01f174a135547c32bea9268379ef33f8125eae251a519dac8f Oct 30 21:45:22.884153 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:22.884123 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx"] Oct 30 21:45:22.884282 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:45:22.884171 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6960e4e7_1cd0_48b0_85f8_c88e46102ad4.slice/crio-1ae8a0f8ed3db4c70def66a15196f155d32ce39a2dd133cf37605cf1464d2f5e WatchSource:0}: Error finding container 1ae8a0f8ed3db4c70def66a15196f155d32ce39a2dd133cf37605cf1464d2f5e: Status 404 returned error can't find the container with id 1ae8a0f8ed3db4c70def66a15196f155d32ce39a2dd133cf37605cf1464d2f5e Oct 30 21:45:22.887166 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:22.887135 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:45:22.887249 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:22.887202 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:45:22.887249 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:22.887235 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:45:22.932375 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:22.932346 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a488c3f-54tr8" event={"ID":"3880a175-d299-45eb-a49a-a89b079b67e7","Type":"ContainerStarted","Data":"e2c5d989e15e098e8f8d95f8edbf3789365b4f44695ba40efd92b4cc4cfeefd4"} Oct 30 21:45:22.932662 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:22.932395 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a488c3f-54tr8" event={"ID":"3880a175-d299-45eb-a49a-a89b079b67e7","Type":"ContainerStarted","Data":"098db84ceb7ccd01f174a135547c32bea9268379ef33f8125eae251a519dac8f"} Oct 30 21:45:22.933744 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:22.933717 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx" event={"ID":"6960e4e7-1cd0-48b0-85f8-c88e46102ad4","Type":"ContainerStarted","Data":"1ae8a0f8ed3db4c70def66a15196f155d32ce39a2dd133cf37605cf1464d2f5e"} Oct 30 21:45:22.935362 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:22.935342 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e7250d4-jggkz" event={"ID":"097d56ed-7515-4b94-9176-56e1c639d24c","Type":"ContainerStarted","Data":"5f5a3df7f6c0ed0c562b7df312cdfb9b272856555cd48c0297c61ece42ad4d4a"} Oct 30 21:45:22.935455 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:22.935387 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e7250d4-jggkz" event={"ID":"097d56ed-7515-4b94-9176-56e1c639d24c","Type":"ContainerStarted","Data":"03b6bb7a2789888e7bca443cded06603985076bce30f960eabe64d9c4f153ece"} Oct 30 21:45:23.054967 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.054911 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e7250d4-jggkz" podStartSLOduration=2.054882074 podStartE2EDuration="2.054882074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:23.052663886 +0000 UTC m=+1306.795184822" watchObservedRunningTime="2026-10-30 21:45:23.054882074 +0000 UTC m=+1306.797403008" Oct 30 21:45:23.594664 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.594629 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9"] Oct 30 21:45:23.598845 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.598827 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9" Oct 30 21:45:23.633609 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.633565 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s"] Oct 30 21:45:23.638294 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.638272 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s" Oct 30 21:45:23.664544 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.664514 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9"] Oct 30 21:45:23.684704 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.684665 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab385653-5874-4072-a986-201ae88230cd-istio-data\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9\" (UID: \"ab385653-5874-4072-a986-201ae88230cd\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9" Oct 30 21:45:23.684704 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.684703 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab385653-5874-4072-a986-201ae88230cd-workload-socket\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9\" (UID: \"ab385653-5874-4072-a986-201ae88230cd\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9" Oct 30 21:45:23.684886 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.684723 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab385653-5874-4072-a986-201ae88230cd-workload-certs\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9\" (UID: \"ab385653-5874-4072-a986-201ae88230cd\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9" Oct 30 21:45:23.684886 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.684797 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-istio-envoy\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s\" (UID: \"93b2f11c-31c6-49c6-a033-c79eaaa4f7c1\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s" Oct 30 21:45:23.684886 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.684847 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-workload-certs\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s\" (UID: \"93b2f11c-31c6-49c6-a033-c79eaaa4f7c1\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s" Oct 30 21:45:23.685029 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.684981 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-istio-data\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s\" (UID: \"93b2f11c-31c6-49c6-a033-c79eaaa4f7c1\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s" Oct 30 21:45:23.685029 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.685001 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab385653-5874-4072-a986-201ae88230cd-istiod-ca-cert\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9\" (UID: \"ab385653-5874-4072-a986-201ae88230cd\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9" Oct 30 21:45:23.685115 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.685040 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab385653-5874-4072-a986-201ae88230cd-istio-podinfo\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9\" (UID: \"ab385653-5874-4072-a986-201ae88230cd\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9" Oct 30 21:45:23.685172 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.685136 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp6x2\" (UniqueName: \"kubernetes.io/projected/ab385653-5874-4072-a986-201ae88230cd-kube-api-access-hp6x2\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9\" (UID: \"ab385653-5874-4072-a986-201ae88230cd\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9" Oct 30 21:45:23.685224 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.685191 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-workload-socket\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s\" (UID: \"93b2f11c-31c6-49c6-a033-c79eaaa4f7c1\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s" Oct 30 21:45:23.685275 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.685263 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-istiod-ca-cert\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s\" (UID: \"93b2f11c-31c6-49c6-a033-c79eaaa4f7c1\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s" Oct 30 21:45:23.685384 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.685362 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-istio-podinfo\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s\" (UID: \"93b2f11c-31c6-49c6-a033-c79eaaa4f7c1\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s" Oct 30 21:45:23.685448 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.685409 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsmdv\" (UniqueName: \"kubernetes.io/projected/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-kube-api-access-jsmdv\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s\" (UID: \"93b2f11c-31c6-49c6-a033-c79eaaa4f7c1\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s" Oct 30 21:45:23.685499 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.685472 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-credential-socket\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s\" (UID: \"93b2f11c-31c6-49c6-a033-c79eaaa4f7c1\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s" Oct 30 21:45:23.685547 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.685521 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-istio-token\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s\" (UID: \"93b2f11c-31c6-49c6-a033-c79eaaa4f7c1\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s" Oct 30 21:45:23.685580 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.685559 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab385653-5874-4072-a986-201ae88230cd-credential-socket\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9\" (UID: \"ab385653-5874-4072-a986-201ae88230cd\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9" Oct 30 21:45:23.685620 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.685589 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab385653-5874-4072-a986-201ae88230cd-istio-token\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9\" (UID: \"ab385653-5874-4072-a986-201ae88230cd\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9" Oct 30 21:45:23.685709 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.685695 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab385653-5874-4072-a986-201ae88230cd-istio-envoy\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9\" (UID: \"ab385653-5874-4072-a986-201ae88230cd\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9" Oct 30 21:45:23.741422 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.741380 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s"] Oct 30 21:45:23.786892 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.786798 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-workload-socket\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s\" (UID: \"93b2f11c-31c6-49c6-a033-c79eaaa4f7c1\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s" Oct 30 21:45:23.786892 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.786857 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-istiod-ca-cert\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s\" (UID: \"93b2f11c-31c6-49c6-a033-c79eaaa4f7c1\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s" Oct 30 21:45:23.787165 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.786931 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-istio-podinfo\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s\" (UID: \"93b2f11c-31c6-49c6-a033-c79eaaa4f7c1\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s" Oct 30 21:45:23.787165 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.786955 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsmdv\" (UniqueName: \"kubernetes.io/projected/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-kube-api-access-jsmdv\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s\" (UID: \"93b2f11c-31c6-49c6-a033-c79eaaa4f7c1\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s" Oct 30 21:45:23.787165 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.786993 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-credential-socket\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s\" (UID: \"93b2f11c-31c6-49c6-a033-c79eaaa4f7c1\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s" Oct 30 21:45:23.787165 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.787028 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-istio-token\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s\" (UID: \"93b2f11c-31c6-49c6-a033-c79eaaa4f7c1\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s" Oct 30 21:45:23.787165 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.787046 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab385653-5874-4072-a986-201ae88230cd-credential-socket\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9\" (UID: \"ab385653-5874-4072-a986-201ae88230cd\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9" Oct 30 21:45:23.787165 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.787063 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab385653-5874-4072-a986-201ae88230cd-istio-token\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9\" (UID: \"ab385653-5874-4072-a986-201ae88230cd\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9" Oct 30 21:45:23.787165 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.787131 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab385653-5874-4072-a986-201ae88230cd-istio-envoy\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9\" (UID: \"ab385653-5874-4072-a986-201ae88230cd\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9" Oct 30 21:45:23.787511 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.787174 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab385653-5874-4072-a986-201ae88230cd-istio-data\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9\" (UID: \"ab385653-5874-4072-a986-201ae88230cd\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9" Oct 30 21:45:23.787511 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.787203 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab385653-5874-4072-a986-201ae88230cd-workload-socket\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9\" (UID: \"ab385653-5874-4072-a986-201ae88230cd\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9" Oct 30 21:45:23.787511 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.787229 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab385653-5874-4072-a986-201ae88230cd-workload-certs\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9\" (UID: \"ab385653-5874-4072-a986-201ae88230cd\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9" Oct 30 21:45:23.787511 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.787253 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-istio-envoy\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s\" (UID: \"93b2f11c-31c6-49c6-a033-c79eaaa4f7c1\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s" Oct 30 21:45:23.787511 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.787283 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-workload-certs\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s\" (UID: \"93b2f11c-31c6-49c6-a033-c79eaaa4f7c1\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s" Oct 30 21:45:23.787511 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.787391 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-istio-data\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s\" (UID: \"93b2f11c-31c6-49c6-a033-c79eaaa4f7c1\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s" Oct 30 21:45:23.787511 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.787417 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab385653-5874-4072-a986-201ae88230cd-istiod-ca-cert\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9\" (UID: \"ab385653-5874-4072-a986-201ae88230cd\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9" Oct 30 21:45:23.787511 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.787432 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab385653-5874-4072-a986-201ae88230cd-credential-socket\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9\" (UID: \"ab385653-5874-4072-a986-201ae88230cd\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9" Oct 30 21:45:23.787511 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.787500 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-credential-socket\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s\" (UID: \"93b2f11c-31c6-49c6-a033-c79eaaa4f7c1\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s" Oct 30 21:45:23.787511 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.787508 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab385653-5874-4072-a986-201ae88230cd-istio-podinfo\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9\" (UID: \"ab385653-5874-4072-a986-201ae88230cd\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9" Oct 30 21:45:23.788024 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.787561 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp6x2\" (UniqueName: \"kubernetes.io/projected/ab385653-5874-4072-a986-201ae88230cd-kube-api-access-hp6x2\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9\" (UID: \"ab385653-5874-4072-a986-201ae88230cd\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9" Oct 30 21:45:23.788024 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.787626 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab385653-5874-4072-a986-201ae88230cd-istio-data\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9\" (UID: \"ab385653-5874-4072-a986-201ae88230cd\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9" Oct 30 21:45:23.788024 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.787669 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-istiod-ca-cert\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s\" (UID: \"93b2f11c-31c6-49c6-a033-c79eaaa4f7c1\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s" Oct 30 21:45:23.788024 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.787949 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab385653-5874-4072-a986-201ae88230cd-workload-socket\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9\" (UID: \"ab385653-5874-4072-a986-201ae88230cd\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9" Oct 30 21:45:23.788228 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.788079 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab385653-5874-4072-a986-201ae88230cd-workload-certs\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9\" (UID: \"ab385653-5874-4072-a986-201ae88230cd\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9" Oct 30 21:45:23.788228 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.788117 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab385653-5874-4072-a986-201ae88230cd-istiod-ca-cert\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9\" (UID: \"ab385653-5874-4072-a986-201ae88230cd\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9" Oct 30 21:45:23.788228 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.788133 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-workload-socket\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s\" (UID: \"93b2f11c-31c6-49c6-a033-c79eaaa4f7c1\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s" Oct 30 21:45:23.788335 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.788279 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-istio-data\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s\" (UID: \"93b2f11c-31c6-49c6-a033-c79eaaa4f7c1\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s" Oct 30 21:45:23.788335 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.788304 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-workload-certs\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s\" (UID: \"93b2f11c-31c6-49c6-a033-c79eaaa4f7c1\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s" Oct 30 21:45:23.789996 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.789963 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab385653-5874-4072-a986-201ae88230cd-istio-envoy\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9\" (UID: \"ab385653-5874-4072-a986-201ae88230cd\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9" Oct 30 21:45:23.790158 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.790138 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-istio-envoy\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s\" (UID: \"93b2f11c-31c6-49c6-a033-c79eaaa4f7c1\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s" Oct 30 21:45:23.790269 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.790252 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab385653-5874-4072-a986-201ae88230cd-istio-podinfo\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9\" (UID: \"ab385653-5874-4072-a986-201ae88230cd\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9" Oct 30 21:45:23.790377 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.790360 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-istio-podinfo\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s\" (UID: \"93b2f11c-31c6-49c6-a033-c79eaaa4f7c1\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s" Oct 30 21:45:23.851887 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.851857 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-istio-token\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s\" (UID: \"93b2f11c-31c6-49c6-a033-c79eaaa4f7c1\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s" Oct 30 21:45:23.852155 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.852115 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab385653-5874-4072-a986-201ae88230cd-istio-token\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9\" (UID: \"ab385653-5874-4072-a986-201ae88230cd\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9" Oct 30 21:45:23.852289 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.852240 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsmdv\" (UniqueName: \"kubernetes.io/projected/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-kube-api-access-jsmdv\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s\" (UID: \"93b2f11c-31c6-49c6-a033-c79eaaa4f7c1\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s" Oct 30 21:45:23.852878 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.852851 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp6x2\" (UniqueName: \"kubernetes.io/projected/ab385653-5874-4072-a986-201ae88230cd-kube-api-access-hp6x2\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9\" (UID: \"ab385653-5874-4072-a986-201ae88230cd\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9" Oct 30 21:45:23.854692 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.854667 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cae3de43-x2ltk"] Oct 30 21:45:23.860292 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.860273 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae3de43-x2ltk" Oct 30 21:45:23.928275 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.928194 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9" Oct 30 21:45:23.931309 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.931240 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cae3de43-x2ltk"] Oct 30 21:45:23.948073 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.948043 5748 generic.go:356] "Generic (PLEG): container finished" podID="097d56ed-7515-4b94-9176-56e1c639d24c" containerID="5f5a3df7f6c0ed0c562b7df312cdfb9b272856555cd48c0297c61ece42ad4d4a" exitCode=0 Oct 30 21:45:23.948481 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.948203 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e7250d4-jggkz" event={"ID":"097d56ed-7515-4b94-9176-56e1c639d24c","Type":"ContainerDied","Data":"5f5a3df7f6c0ed0c562b7df312cdfb9b272856555cd48c0297c61ece42ad4d4a"} Oct 30 21:45:23.950109 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.950080 5748 generic.go:356] "Generic (PLEG): container finished" podID="3880a175-d299-45eb-a49a-a89b079b67e7" containerID="e2c5d989e15e098e8f8d95f8edbf3789365b4f44695ba40efd92b4cc4cfeefd4" exitCode=0 Oct 30 21:45:23.950222 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.950172 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a488c3f-54tr8" event={"ID":"3880a175-d299-45eb-a49a-a89b079b67e7","Type":"ContainerDied","Data":"e2c5d989e15e098e8f8d95f8edbf3789365b4f44695ba40efd92b4cc4cfeefd4"} Oct 30 21:45:23.951974 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.951950 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx" event={"ID":"6960e4e7-1cd0-48b0-85f8-c88e46102ad4","Type":"ContainerStarted","Data":"7d2f4ae4b42ef33ebb77c2f7e1bb6c474a574bcf5935639fdebe2ad105c08deb"} Oct 30 21:45:23.952845 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.952824 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s" Oct 30 21:45:23.994209 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.989970 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d77664f-6bf2-4f3a-8f07-490f1d4c12b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cae3de43-x2ltk\" (UID: \"0d77664f-6bf2-4f3a-8f07-490f1d4c12b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae3de43-x2ltk" Oct 30 21:45:23.994209 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.990213 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d77664f-6bf2-4f3a-8f07-490f1d4c12b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cae3de43-x2ltk\" (UID: \"0d77664f-6bf2-4f3a-8f07-490f1d4c12b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae3de43-x2ltk" Oct 30 21:45:23.994209 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:23.990417 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqwlp\" (UniqueName: \"kubernetes.io/projected/0d77664f-6bf2-4f3a-8f07-490f1d4c12b7-kube-api-access-qqwlp\") pod \"maas-api-revoke-sub-cae3de43-x2ltk\" (UID: \"0d77664f-6bf2-4f3a-8f07-490f1d4c12b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae3de43-x2ltk" Oct 30 21:45:24.092479 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:24.092451 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d77664f-6bf2-4f3a-8f07-490f1d4c12b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cae3de43-x2ltk\" (UID: \"0d77664f-6bf2-4f3a-8f07-490f1d4c12b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae3de43-x2ltk" Oct 30 21:45:24.092649 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:24.092611 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d77664f-6bf2-4f3a-8f07-490f1d4c12b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cae3de43-x2ltk\" (UID: \"0d77664f-6bf2-4f3a-8f07-490f1d4c12b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae3de43-x2ltk" Oct 30 21:45:24.093136 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:24.093114 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqwlp\" (UniqueName: \"kubernetes.io/projected/0d77664f-6bf2-4f3a-8f07-490f1d4c12b7-kube-api-access-qqwlp\") pod \"maas-api-revoke-sub-cae3de43-x2ltk\" (UID: \"0d77664f-6bf2-4f3a-8f07-490f1d4c12b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae3de43-x2ltk" Oct 30 21:45:24.093420 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:24.093383 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d77664f-6bf2-4f3a-8f07-490f1d4c12b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cae3de43-x2ltk\" (UID: \"0d77664f-6bf2-4f3a-8f07-490f1d4c12b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae3de43-x2ltk" Oct 30 21:45:24.094975 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:24.094955 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d77664f-6bf2-4f3a-8f07-490f1d4c12b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cae3de43-x2ltk\" (UID: \"0d77664f-6bf2-4f3a-8f07-490f1d4c12b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae3de43-x2ltk" Oct 30 21:45:24.155161 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:24.155132 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqwlp\" (UniqueName: \"kubernetes.io/projected/0d77664f-6bf2-4f3a-8f07-490f1d4c12b7-kube-api-access-qqwlp\") pod \"maas-api-revoke-sub-cae3de43-x2ltk\" (UID: \"0d77664f-6bf2-4f3a-8f07-490f1d4c12b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae3de43-x2ltk" Oct 30 21:45:24.187326 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:24.187299 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae3de43-x2ltk" Oct 30 21:45:24.371745 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:24.371645 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx" podStartSLOduration=2.371625944 podStartE2EDuration="2.371625944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:24.368808947 +0000 UTC m=+1308.111329883" watchObservedRunningTime="2026-10-30 21:45:24.371625944 +0000 UTC m=+1308.114146882" Oct 30 21:45:24.431367 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:45:24.431330 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93b2f11c_31c6_49c6_a033_c79eaaa4f7c1.slice/crio-b19be204e900ef747036674545d0d1258756b739a6ec3de1c00c4770e46cabff WatchSource:0}: Error finding container b19be204e900ef747036674545d0d1258756b739a6ec3de1c00c4770e46cabff: Status 404 returned error can't find the container with id b19be204e900ef747036674545d0d1258756b739a6ec3de1c00c4770e46cabff Oct 30 21:45:24.441277 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:24.441230 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:45:24.441399 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:24.441379 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:45:24.441590 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:24.441571 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:45:24.441798 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:24.441772 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:45:24.442086 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:24.442060 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:45:24.442876 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:24.442834 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:45:24.451168 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:24.448444 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s"] Oct 30 21:45:24.451168 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:24.448517 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9"] Oct 30 21:45:24.519160 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:24.503404 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx" Oct 30 21:45:24.527026 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:24.526036 5748 patch_prober.go:29] interesting pod/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 21:45:24.527026 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:24.526108 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx" podUID="6960e4e7-1cd0-48b0-85f8-c88e46102ad4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 21:45:24.605581 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:24.605391 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cae3de43-x2ltk"] Oct 30 21:45:24.962285 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:24.962251 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9" event={"ID":"ab385653-5874-4072-a986-201ae88230cd","Type":"ContainerStarted","Data":"3e74f296bfde4cad229c4a81110a392480109903ea53de450cae815458a8059c"} Oct 30 21:45:24.962695 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:24.962314 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9" event={"ID":"ab385653-5874-4072-a986-201ae88230cd","Type":"ContainerStarted","Data":"c1b49119b77c42d604c2d78eb49d0c861276965995f9e5c8051d0efaf0f1f64f"} Oct 30 21:45:24.964660 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:24.964627 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s" event={"ID":"93b2f11c-31c6-49c6-a033-c79eaaa4f7c1","Type":"ContainerStarted","Data":"b132b953533742f5afb46291f61a3a6f551a2a64903c5f5c1b50e1f9abcdbcd8"} Oct 30 21:45:24.964789 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:24.964673 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s" event={"ID":"93b2f11c-31c6-49c6-a033-c79eaaa4f7c1","Type":"ContainerStarted","Data":"b19be204e900ef747036674545d0d1258756b739a6ec3de1c00c4770e46cabff"} Oct 30 21:45:24.966029 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:24.965997 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae3de43-x2ltk" event={"ID":"0d77664f-6bf2-4f3a-8f07-490f1d4c12b7","Type":"ContainerStarted","Data":"ce1399619f049b308c19eaaab0d3c194db036833b57f7349d050e3f1bea953a0"} Oct 30 21:45:25.091411 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:25.091360 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9" podStartSLOduration=2.091344733 podStartE2EDuration="2.091344733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:25.088002652 +0000 UTC m=+1308.830523615" watchObservedRunningTime="2026-10-30 21:45:25.091344733 +0000 UTC m=+1308.833865668" Oct 30 21:45:25.154961 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:25.154330 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6dc779f-zzcqb"] Oct 30 21:45:25.165740 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:25.165693 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6dc779f-zzcqb" Oct 30 21:45:25.393799 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:25.393707 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d335b145-924d-41c8-852f-2ebcd75c6e53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6dc779f-zzcqb\" (UID: \"d335b145-924d-41c8-852f-2ebcd75c6e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6dc779f-zzcqb" Oct 30 21:45:25.393981 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:25.393836 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk9ww\" (UniqueName: \"kubernetes.io/projected/d335b145-924d-41c8-852f-2ebcd75c6e53-kube-api-access-sk9ww\") pod \"maas-api-revoke-sub-b6dc779f-zzcqb\" (UID: \"d335b145-924d-41c8-852f-2ebcd75c6e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6dc779f-zzcqb" Oct 30 21:45:25.394081 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:25.394005 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d335b145-924d-41c8-852f-2ebcd75c6e53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6dc779f-zzcqb\" (UID: \"d335b145-924d-41c8-852f-2ebcd75c6e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6dc779f-zzcqb" Oct 30 21:45:25.397970 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:25.397940 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6dc779f-zzcqb"] Oct 30 21:45:25.423924 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:25.410407 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s" podStartSLOduration=2.410386776 podStartE2EDuration="2.410386776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:25.391316076 +0000 UTC m=+1309.133837015" watchObservedRunningTime="2026-10-30 21:45:25.410386776 +0000 UTC m=+1309.152907714" Oct 30 21:45:25.495798 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:25.495759 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d335b145-924d-41c8-852f-2ebcd75c6e53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6dc779f-zzcqb\" (UID: \"d335b145-924d-41c8-852f-2ebcd75c6e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6dc779f-zzcqb" Oct 30 21:45:25.496145 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:25.496128 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk9ww\" (UniqueName: \"kubernetes.io/projected/d335b145-924d-41c8-852f-2ebcd75c6e53-kube-api-access-sk9ww\") pod \"maas-api-revoke-sub-b6dc779f-zzcqb\" (UID: \"d335b145-924d-41c8-852f-2ebcd75c6e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6dc779f-zzcqb" Oct 30 21:45:25.496387 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:25.496371 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d335b145-924d-41c8-852f-2ebcd75c6e53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6dc779f-zzcqb\" (UID: \"d335b145-924d-41c8-852f-2ebcd75c6e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6dc779f-zzcqb" Oct 30 21:45:25.498279 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:25.497591 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d335b145-924d-41c8-852f-2ebcd75c6e53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6dc779f-zzcqb\" (UID: \"d335b145-924d-41c8-852f-2ebcd75c6e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6dc779f-zzcqb" Oct 30 21:45:25.503621 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:25.503221 5748 patch_prober.go:29] interesting pod/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 21:45:25.503621 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:25.503271 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx" podUID="6960e4e7-1cd0-48b0-85f8-c88e46102ad4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 21:45:25.550972 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:25.550809 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a488c3f-54tr8" Oct 30 21:45:25.551602 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:25.551545 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d335b145-924d-41c8-852f-2ebcd75c6e53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6dc779f-zzcqb\" (UID: \"d335b145-924d-41c8-852f-2ebcd75c6e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6dc779f-zzcqb" Oct 30 21:45:25.599075 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:25.598911 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk9ww\" (UniqueName: \"kubernetes.io/projected/d335b145-924d-41c8-852f-2ebcd75c6e53-kube-api-access-sk9ww\") pod \"maas-api-revoke-sub-b6dc779f-zzcqb\" (UID: \"d335b145-924d-41c8-852f-2ebcd75c6e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6dc779f-zzcqb" Oct 30 21:45:25.692933 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:25.689871 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e7250d4-jggkz" Oct 30 21:45:25.735918 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:25.723326 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/097d56ed-7515-4b94-9176-56e1c639d24c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/097d56ed-7515-4b94-9176-56e1c639d24c-maas-api-xks-ca\") pod \"097d56ed-7515-4b94-9176-56e1c639d24c\" (UID: \"097d56ed-7515-4b94-9176-56e1c639d24c\") " Oct 30 21:45:25.735918 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:25.723775 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3880a175-d299-45eb-a49a-a89b079b67e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3880a175-d299-45eb-a49a-a89b079b67e7-maas-api-openshift-ca\") pod \"3880a175-d299-45eb-a49a-a89b079b67e7\" (UID: \"3880a175-d299-45eb-a49a-a89b079b67e7\") " Oct 30 21:45:25.735918 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:25.723826 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3880a175-d299-45eb-a49a-a89b079b67e7-kube-api-access-nctks\" (UniqueName: \"kubernetes.io/projected/3880a175-d299-45eb-a49a-a89b079b67e7-kube-api-access-nctks\") pod \"3880a175-d299-45eb-a49a-a89b079b67e7\" (UID: \"3880a175-d299-45eb-a49a-a89b079b67e7\") " Oct 30 21:45:25.735918 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:25.728256 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3880a175-d299-45eb-a49a-a89b079b67e7-maas-api-openshift-ca" pod "3880a175-d299-45eb-a49a-a89b079b67e7" (UID: "3880a175-d299-45eb-a49a-a89b079b67e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:25.735918 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:25.731007 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/097d56ed-7515-4b94-9176-56e1c639d24c-kube-api-access-zrmzz\" (UniqueName: \"kubernetes.io/projected/097d56ed-7515-4b94-9176-56e1c639d24c-kube-api-access-zrmzz\") pod \"097d56ed-7515-4b94-9176-56e1c639d24c\" (UID: \"097d56ed-7515-4b94-9176-56e1c639d24c\") " Oct 30 21:45:25.735918 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:25.731294 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3880a175-d299-45eb-a49a-a89b079b67e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3880a175-d299-45eb-a49a-a89b079b67e7-maas-api-xks-ca\") pod \"3880a175-d299-45eb-a49a-a89b079b67e7\" (UID: \"3880a175-d299-45eb-a49a-a89b079b67e7\") " Oct 30 21:45:25.735918 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:25.731604 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/097d56ed-7515-4b94-9176-56e1c639d24c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/097d56ed-7515-4b94-9176-56e1c639d24c-maas-api-openshift-ca\") pod \"097d56ed-7515-4b94-9176-56e1c639d24c\" (UID: \"097d56ed-7515-4b94-9176-56e1c639d24c\") " Oct 30 21:45:25.742783 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:25.742445 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3880a175-d299-45eb-a49a-a89b079b67e7-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:25.751194 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:25.751141 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/097d56ed-7515-4b94-9176-56e1c639d24c-maas-api-openshift-ca" pod "097d56ed-7515-4b94-9176-56e1c639d24c" (UID: "097d56ed-7515-4b94-9176-56e1c639d24c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:25.784300 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:25.784259 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/097d56ed-7515-4b94-9176-56e1c639d24c-maas-api-xks-ca" pod "097d56ed-7515-4b94-9176-56e1c639d24c" (UID: "097d56ed-7515-4b94-9176-56e1c639d24c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:45:25.787688 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:25.787647 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3880a175-d299-45eb-a49a-a89b079b67e7-maas-api-xks-ca" pod "3880a175-d299-45eb-a49a-a89b079b67e7" (UID: "3880a175-d299-45eb-a49a-a89b079b67e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:45:25.787688 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:25.787675 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3880a175-d299-45eb-a49a-a89b079b67e7-kube-api-access-nctks" pod "3880a175-d299-45eb-a49a-a89b079b67e7" (UID: "3880a175-d299-45eb-a49a-a89b079b67e7"). InnerVolumeSpecName "kube-api-access-nctks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:25.787946 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:25.787732 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/097d56ed-7515-4b94-9176-56e1c639d24c-kube-api-access-zrmzz" pod "097d56ed-7515-4b94-9176-56e1c639d24c" (UID: "097d56ed-7515-4b94-9176-56e1c639d24c"). InnerVolumeSpecName "kube-api-access-zrmzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:25.796122 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:25.796084 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6dc779f-zzcqb" Oct 30 21:45:25.877221 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:25.877181 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/097d56ed-7515-4b94-9176-56e1c639d24c-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:25.877438 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:25.877418 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nctks\" (UniqueName: \"kubernetes.io/projected/3880a175-d299-45eb-a49a-a89b079b67e7-kube-api-access-nctks\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:25.877566 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:25.877552 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrmzz\" (UniqueName: \"kubernetes.io/projected/097d56ed-7515-4b94-9176-56e1c639d24c-kube-api-access-zrmzz\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:25.877649 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:25.877637 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3880a175-d299-45eb-a49a-a89b079b67e7-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:25.877735 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:25.877723 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/097d56ed-7515-4b94-9176-56e1c639d24c-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:25.933359 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:25.933323 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9" Oct 30 21:45:25.954293 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:25.954257 5748 patch_prober.go:29] interesting pod/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 21:45:25.954587 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:25.954555 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9" podUID="ab385653-5874-4072-a986-201ae88230cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 21:45:25.956004 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:25.955981 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s" Oct 30 21:45:25.990166 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:25.990137 5748 patch_prober.go:29] interesting pod/e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 21:45:25.990621 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:25.990595 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s" podUID="93b2f11c-31c6-49c6-a033-c79eaaa4f7c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 21:45:26.112990 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:26.096186 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e7250d4-jggkz" event={"ID":"097d56ed-7515-4b94-9176-56e1c639d24c","Type":"ContainerDied","Data":"03b6bb7a2789888e7bca443cded06603985076bce30f960eabe64d9c4f153ece"} Oct 30 21:45:26.112990 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:26.096248 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03b6bb7a2789888e7bca443cded06603985076bce30f960eabe64d9c4f153ece" Oct 30 21:45:26.112990 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:26.096325 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e7250d4-jggkz" Oct 30 21:45:26.191519 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:26.191490 5748 generic.go:356] "Generic (PLEG): container finished" podID="0d77664f-6bf2-4f3a-8f07-490f1d4c12b7" containerID="a9c24c5cb9181a2d69b1610c858d8910c6444201b11d36d3e01c0c9f38227f1d" exitCode=0 Oct 30 21:45:26.191685 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:26.191615 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae3de43-x2ltk" event={"ID":"0d77664f-6bf2-4f3a-8f07-490f1d4c12b7","Type":"ContainerDied","Data":"a9c24c5cb9181a2d69b1610c858d8910c6444201b11d36d3e01c0c9f38227f1d"} Oct 30 21:45:26.194330 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:26.194295 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a488c3f-54tr8" Oct 30 21:45:26.297140 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:26.262154 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a488c3f-54tr8" event={"ID":"3880a175-d299-45eb-a49a-a89b079b67e7","Type":"ContainerDied","Data":"098db84ceb7ccd01f174a135547c32bea9268379ef33f8125eae251a519dac8f"} Oct 30 21:45:26.297140 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:26.262224 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="098db84ceb7ccd01f174a135547c32bea9268379ef33f8125eae251a519dac8f" Oct 30 21:45:26.564616 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:26.544322 5748 patch_prober.go:29] interesting pod/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 21:45:26.564616 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:26.544389 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx" podUID="6960e4e7-1cd0-48b0-85f8-c88e46102ad4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 21:45:26.787788 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:26.787764 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6dc779f-zzcqb"] Oct 30 21:45:26.932920 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:26.932646 5748 patch_prober.go:29] interesting pod/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 21:45:26.932920 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:26.932724 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9" podUID="ab385653-5874-4072-a986-201ae88230cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 21:45:26.958092 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:26.958054 5748 patch_prober.go:29] interesting pod/e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 21:45:26.958333 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:26.958307 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s" podUID="93b2f11c-31c6-49c6-a033-c79eaaa4f7c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 21:45:27.245986 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:27.245945 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6dc779f-zzcqb" event={"ID":"d335b145-924d-41c8-852f-2ebcd75c6e53","Type":"ContainerStarted","Data":"e5e45804b55d0b2fe1d81c5f142480583d31e73868f9153628fa983958b22e81"} Oct 30 21:45:27.585604 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:27.584268 5748 patch_prober.go:29] interesting pod/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 21:45:27.585604 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:27.584335 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx" podUID="6960e4e7-1cd0-48b0-85f8-c88e46102ad4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 21:45:27.790407 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:27.790373 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae3de43-x2ltk" Oct 30 21:45:27.947977 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:27.947579 5748 patch_prober.go:29] interesting pod/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 21:45:27.947977 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:27.947648 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9" podUID="ab385653-5874-4072-a986-201ae88230cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 21:45:27.970967 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:27.962380 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d77664f-6bf2-4f3a-8f07-490f1d4c12b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d77664f-6bf2-4f3a-8f07-490f1d4c12b7-maas-api-openshift-ca\") pod \"0d77664f-6bf2-4f3a-8f07-490f1d4c12b7\" (UID: \"0d77664f-6bf2-4f3a-8f07-490f1d4c12b7\") " Oct 30 21:45:27.970967 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:27.962481 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d77664f-6bf2-4f3a-8f07-490f1d4c12b7-kube-api-access-qqwlp\" (UniqueName: \"kubernetes.io/projected/0d77664f-6bf2-4f3a-8f07-490f1d4c12b7-kube-api-access-qqwlp\") pod \"0d77664f-6bf2-4f3a-8f07-490f1d4c12b7\" (UID: \"0d77664f-6bf2-4f3a-8f07-490f1d4c12b7\") " Oct 30 21:45:27.970967 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:27.962547 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d77664f-6bf2-4f3a-8f07-490f1d4c12b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d77664f-6bf2-4f3a-8f07-490f1d4c12b7-maas-api-xks-ca\") pod \"0d77664f-6bf2-4f3a-8f07-490f1d4c12b7\" (UID: \"0d77664f-6bf2-4f3a-8f07-490f1d4c12b7\") " Oct 30 21:45:27.984939 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:27.973364 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d77664f-6bf2-4f3a-8f07-490f1d4c12b7-maas-api-openshift-ca" pod "0d77664f-6bf2-4f3a-8f07-490f1d4c12b7" (UID: "0d77664f-6bf2-4f3a-8f07-490f1d4c12b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:27.988151 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:27.988112 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d77664f-6bf2-4f3a-8f07-490f1d4c12b7-kube-api-access-qqwlp" pod "0d77664f-6bf2-4f3a-8f07-490f1d4c12b7" (UID: "0d77664f-6bf2-4f3a-8f07-490f1d4c12b7"). InnerVolumeSpecName "kube-api-access-qqwlp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:27.990417 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:27.990386 5748 patch_prober.go:29] interesting pod/e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 21:45:27.990533 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:27.990456 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s" podUID="93b2f11c-31c6-49c6-a033-c79eaaa4f7c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 21:45:27.992063 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:27.991999 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d77664f-6bf2-4f3a-8f07-490f1d4c12b7-maas-api-xks-ca" pod "0d77664f-6bf2-4f3a-8f07-490f1d4c12b7" (UID: "0d77664f-6bf2-4f3a-8f07-490f1d4c12b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:45:28.094958 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:28.067097 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d77664f-6bf2-4f3a-8f07-490f1d4c12b7-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:28.094958 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:28.094171 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqwlp\" (UniqueName: \"kubernetes.io/projected/0d77664f-6bf2-4f3a-8f07-490f1d4c12b7-kube-api-access-qqwlp\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:28.094958 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:28.094216 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d77664f-6bf2-4f3a-8f07-490f1d4c12b7-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:28.343980 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:28.343849 5748 generic.go:356] "Generic (PLEG): container finished" podID="d335b145-924d-41c8-852f-2ebcd75c6e53" containerID="5d97bab635960b2e303e4e9b9f1e67a318514ef26d38aab60dbc218a21d45148" exitCode=0 Oct 30 21:45:28.344869 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:28.344835 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6dc779f-zzcqb" event={"ID":"d335b145-924d-41c8-852f-2ebcd75c6e53","Type":"ContainerDied","Data":"5d97bab635960b2e303e4e9b9f1e67a318514ef26d38aab60dbc218a21d45148"} Oct 30 21:45:28.355773 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:28.355737 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae3de43-x2ltk" event={"ID":"0d77664f-6bf2-4f3a-8f07-490f1d4c12b7","Type":"ContainerDied","Data":"ce1399619f049b308c19eaaab0d3c194db036833b57f7349d050e3f1bea953a0"} Oct 30 21:45:28.360076 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:28.360047 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce1399619f049b308c19eaaab0d3c194db036833b57f7349d050e3f1bea953a0" Oct 30 21:45:28.360311 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:28.360296 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae3de43-x2ltk" Oct 30 21:45:28.510922 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:28.505233 5748 patch_prober.go:29] interesting pod/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 21:45:28.510922 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:28.505290 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx" podUID="6960e4e7-1cd0-48b0-85f8-c88e46102ad4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 21:45:28.853922 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:28.853178 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32ad80cc-cqmzb"] Oct 30 21:45:28.858923 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:28.854175 5748 cpu_manager.go:414] "containerMap: removing container" podUID="097d56ed-7515-4b94-9176-56e1c639d24c" containerName="revoke-keys" Oct 30 21:45:28.858923 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:28.854202 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="097d56ed-7515-4b94-9176-56e1c639d24c" containerName="revoke-keys" Oct 30 21:45:28.858923 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:28.854234 5748 cpu_manager.go:414] "containerMap: removing container" podUID="0d77664f-6bf2-4f3a-8f07-490f1d4c12b7" containerName="revoke-keys" Oct 30 21:45:28.858923 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:28.854242 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d77664f-6bf2-4f3a-8f07-490f1d4c12b7" containerName="revoke-keys" Oct 30 21:45:28.858923 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:28.854272 5748 cpu_manager.go:414] "containerMap: removing container" podUID="3880a175-d299-45eb-a49a-a89b079b67e7" containerName="revoke-keys" Oct 30 21:45:28.858923 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:28.854281 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3880a175-d299-45eb-a49a-a89b079b67e7" containerName="revoke-keys" Oct 30 21:45:28.858923 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:28.854430 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="097d56ed-7515-4b94-9176-56e1c639d24c" containerName="revoke-keys" Oct 30 21:45:28.858923 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:28.854461 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d77664f-6bf2-4f3a-8f07-490f1d4c12b7" containerName="revoke-keys" Oct 30 21:45:28.858923 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:28.854476 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="3880a175-d299-45eb-a49a-a89b079b67e7" containerName="revoke-keys" Oct 30 21:45:28.878924 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:28.876215 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ad80cc-cqmzb" Oct 30 21:45:28.887490 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:28.887392 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32ad80cc-cqmzb"] Oct 30 21:45:28.986377 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:28.986336 5748 patch_prober.go:29] interesting pod/e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 21:45:28.986583 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:28.986399 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s" podUID="93b2f11c-31c6-49c6-a033-c79eaaa4f7c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 21:45:28.986583 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:28.986344 5748 patch_prober.go:29] interesting pod/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 21:45:28.986583 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:28.986471 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9" podUID="ab385653-5874-4072-a986-201ae88230cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 21:45:28.987640 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:28.987607 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99cf5ffb-5049-4f6b-aca5-4f6b22c55483-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32ad80cc-cqmzb\" (UID: \"99cf5ffb-5049-4f6b-aca5-4f6b22c55483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ad80cc-cqmzb" Oct 30 21:45:28.987760 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:28.987662 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx7wf\" (UniqueName: \"kubernetes.io/projected/99cf5ffb-5049-4f6b-aca5-4f6b22c55483-kube-api-access-lx7wf\") pod \"maas-api-revoke-sub-32ad80cc-cqmzb\" (UID: \"99cf5ffb-5049-4f6b-aca5-4f6b22c55483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ad80cc-cqmzb" Oct 30 21:45:28.987760 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:28.987718 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99cf5ffb-5049-4f6b-aca5-4f6b22c55483-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32ad80cc-cqmzb\" (UID: \"99cf5ffb-5049-4f6b-aca5-4f6b22c55483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ad80cc-cqmzb" Oct 30 21:45:29.089296 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:29.089262 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99cf5ffb-5049-4f6b-aca5-4f6b22c55483-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32ad80cc-cqmzb\" (UID: \"99cf5ffb-5049-4f6b-aca5-4f6b22c55483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ad80cc-cqmzb" Oct 30 21:45:29.089521 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:29.089503 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx7wf\" (UniqueName: \"kubernetes.io/projected/99cf5ffb-5049-4f6b-aca5-4f6b22c55483-kube-api-access-lx7wf\") pod \"maas-api-revoke-sub-32ad80cc-cqmzb\" (UID: \"99cf5ffb-5049-4f6b-aca5-4f6b22c55483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ad80cc-cqmzb" Oct 30 21:45:29.089690 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:29.089669 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99cf5ffb-5049-4f6b-aca5-4f6b22c55483-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32ad80cc-cqmzb\" (UID: \"99cf5ffb-5049-4f6b-aca5-4f6b22c55483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ad80cc-cqmzb" Oct 30 21:45:29.090581 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:29.090198 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99cf5ffb-5049-4f6b-aca5-4f6b22c55483-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32ad80cc-cqmzb\" (UID: \"99cf5ffb-5049-4f6b-aca5-4f6b22c55483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ad80cc-cqmzb" Oct 30 21:45:29.092287 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:29.092263 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99cf5ffb-5049-4f6b-aca5-4f6b22c55483-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32ad80cc-cqmzb\" (UID: \"99cf5ffb-5049-4f6b-aca5-4f6b22c55483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ad80cc-cqmzb" Oct 30 21:45:29.166766 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:29.166734 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx7wf\" (UniqueName: \"kubernetes.io/projected/99cf5ffb-5049-4f6b-aca5-4f6b22c55483-kube-api-access-lx7wf\") pod \"maas-api-revoke-sub-32ad80cc-cqmzb\" (UID: \"99cf5ffb-5049-4f6b-aca5-4f6b22c55483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ad80cc-cqmzb" Oct 30 21:45:29.196707 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:29.196673 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ad80cc-cqmzb" Oct 30 21:45:29.507996 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:29.504578 5748 patch_prober.go:29] interesting pod/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 21:45:29.526317 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:29.526274 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx" podUID="6960e4e7-1cd0-48b0-85f8-c88e46102ad4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 21:45:29.773955 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:29.773871 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32ad80cc-cqmzb"] Oct 30 21:45:29.787420 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:29.787385 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6dc779f-zzcqb" Oct 30 21:45:29.798947 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:29.798786 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d335b145-924d-41c8-852f-2ebcd75c6e53-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d335b145-924d-41c8-852f-2ebcd75c6e53-maas-api-xks-ca\") pod \"d335b145-924d-41c8-852f-2ebcd75c6e53\" (UID: \"d335b145-924d-41c8-852f-2ebcd75c6e53\") " Oct 30 21:45:29.799209 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:29.799187 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d335b145-924d-41c8-852f-2ebcd75c6e53-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d335b145-924d-41c8-852f-2ebcd75c6e53-maas-api-openshift-ca\") pod \"d335b145-924d-41c8-852f-2ebcd75c6e53\" (UID: \"d335b145-924d-41c8-852f-2ebcd75c6e53\") " Oct 30 21:45:29.799570 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:29.799480 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d335b145-924d-41c8-852f-2ebcd75c6e53-maas-api-openshift-ca" pod "d335b145-924d-41c8-852f-2ebcd75c6e53" (UID: "d335b145-924d-41c8-852f-2ebcd75c6e53"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:29.799677 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:29.799592 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d335b145-924d-41c8-852f-2ebcd75c6e53-kube-api-access-sk9ww\" (UniqueName: \"kubernetes.io/projected/d335b145-924d-41c8-852f-2ebcd75c6e53-kube-api-access-sk9ww\") pod \"d335b145-924d-41c8-852f-2ebcd75c6e53\" (UID: \"d335b145-924d-41c8-852f-2ebcd75c6e53\") " Oct 30 21:45:29.807938 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:29.800388 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d335b145-924d-41c8-852f-2ebcd75c6e53-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:29.862134 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:29.862068 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d335b145-924d-41c8-852f-2ebcd75c6e53-maas-api-xks-ca" pod "d335b145-924d-41c8-852f-2ebcd75c6e53" (UID: "d335b145-924d-41c8-852f-2ebcd75c6e53"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:45:29.862134 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:29.862101 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d335b145-924d-41c8-852f-2ebcd75c6e53-kube-api-access-sk9ww" pod "d335b145-924d-41c8-852f-2ebcd75c6e53" (UID: "d335b145-924d-41c8-852f-2ebcd75c6e53"). InnerVolumeSpecName "kube-api-access-sk9ww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:29.904922 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:29.901844 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d335b145-924d-41c8-852f-2ebcd75c6e53-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:29.904922 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:29.901883 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk9ww\" (UniqueName: \"kubernetes.io/projected/d335b145-924d-41c8-852f-2ebcd75c6e53-kube-api-access-sk9ww\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:29.940811 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:29.940311 5748 patch_prober.go:29] interesting pod/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 21:45:29.940811 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:29.940375 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9" podUID="ab385653-5874-4072-a986-201ae88230cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 21:45:29.960933 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:29.960247 5748 patch_prober.go:29] interesting pod/e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 21:45:29.960933 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:29.960307 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s" podUID="93b2f11c-31c6-49c6-a033-c79eaaa4f7c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 21:45:30.196952 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:45:30.196892 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:45:30.387872 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:30.387841 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ad80cc-cqmzb" event={"ID":"99cf5ffb-5049-4f6b-aca5-4f6b22c55483","Type":"ContainerStarted","Data":"d96cd213fa3aa9b038f9f709fa6509d56b70730a3ae818a68770394f13833bed"} Oct 30 21:45:30.388088 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:30.387922 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ad80cc-cqmzb" event={"ID":"99cf5ffb-5049-4f6b-aca5-4f6b22c55483","Type":"ContainerStarted","Data":"005ab48d31061cb1075b732110df963008312e7766cd7a164b2fffc3b11f1577"} Oct 30 21:45:30.389402 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:30.389350 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6dc779f-zzcqb" Oct 30 21:45:30.389840 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:30.389380 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6dc779f-zzcqb" event={"ID":"d335b145-924d-41c8-852f-2ebcd75c6e53","Type":"ContainerDied","Data":"e5e45804b55d0b2fe1d81c5f142480583d31e73868f9153628fa983958b22e81"} Oct 30 21:45:30.389941 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:30.389859 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5e45804b55d0b2fe1d81c5f142480583d31e73868f9153628fa983958b22e81" Oct 30 21:45:30.569922 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:30.569476 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ad80cc-cqmzb" podStartSLOduration=2.569460556 podStartE2EDuration="2.569460556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:30.550747197 +0000 UTC m=+1314.293268123" watchObservedRunningTime="2026-10-30 21:45:30.569460556 +0000 UTC m=+1314.311981491" Oct 30 21:45:30.929211 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:30.929158 5748 patch_prober.go:29] interesting pod/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 21:45:30.929399 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:30.929221 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9" podUID="ab385653-5874-4072-a986-201ae88230cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 21:45:30.932149 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:30.932121 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx" Oct 30 21:45:30.933403 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:30.932872 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx" Oct 30 21:45:30.933721 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:30.933699 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx" Oct 30 21:45:30.954393 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:30.954328 5748 patch_prober.go:29] interesting pod/e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 21:45:30.954559 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:30.954427 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s" podUID="93b2f11c-31c6-49c6-a033-c79eaaa4f7c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 21:45:31.401818 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:31.401778 5748 generic.go:356] "Generic (PLEG): container finished" podID="99cf5ffb-5049-4f6b-aca5-4f6b22c55483" containerID="d96cd213fa3aa9b038f9f709fa6509d56b70730a3ae818a68770394f13833bed" exitCode=0 Oct 30 21:45:31.402015 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:31.401865 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ad80cc-cqmzb" event={"ID":"99cf5ffb-5049-4f6b-aca5-4f6b22c55483","Type":"ContainerDied","Data":"d96cd213fa3aa9b038f9f709fa6509d56b70730a3ae818a68770394f13833bed"} Oct 30 21:45:32.145886 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:32.145855 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9" Oct 30 21:45:32.147116 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:32.147071 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9" Oct 30 21:45:32.147987 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:32.147966 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9" Oct 30 21:45:32.182916 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:32.182866 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s" Oct 30 21:45:32.183364 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:32.183337 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s" Oct 30 21:45:32.184507 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:32.184482 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s" Oct 30 21:45:32.294061 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:32.294028 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a092c8c-ftf9n"] Oct 30 21:45:32.294662 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:32.294646 5748 cpu_manager.go:414] "containerMap: removing container" podUID="d335b145-924d-41c8-852f-2ebcd75c6e53" containerName="revoke-keys" Oct 30 21:45:32.294723 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:32.294663 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d335b145-924d-41c8-852f-2ebcd75c6e53" containerName="revoke-keys" Oct 30 21:45:32.294774 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:32.294765 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="d335b145-924d-41c8-852f-2ebcd75c6e53" containerName="revoke-keys" Oct 30 21:45:32.298438 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:32.298407 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a092c8c-ftf9n" Oct 30 21:45:32.375566 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:32.375525 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a092c8c-ftf9n"] Oct 30 21:45:32.394677 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:32.394633 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k2fz\" (UniqueName: \"kubernetes.io/projected/50e120fc-8f50-41eb-81f2-71e5bc8b74cf-kube-api-access-5k2fz\") pod \"maas-api-revoke-sub-5a092c8c-ftf9n\" (UID: \"50e120fc-8f50-41eb-81f2-71e5bc8b74cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a092c8c-ftf9n" Oct 30 21:45:32.394950 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:32.394922 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50e120fc-8f50-41eb-81f2-71e5bc8b74cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a092c8c-ftf9n\" (UID: \"50e120fc-8f50-41eb-81f2-71e5bc8b74cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a092c8c-ftf9n" Oct 30 21:45:32.395096 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:32.395074 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50e120fc-8f50-41eb-81f2-71e5bc8b74cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a092c8c-ftf9n\" (UID: \"50e120fc-8f50-41eb-81f2-71e5bc8b74cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a092c8c-ftf9n" Oct 30 21:45:32.496810 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:32.496775 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50e120fc-8f50-41eb-81f2-71e5bc8b74cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a092c8c-ftf9n\" (UID: \"50e120fc-8f50-41eb-81f2-71e5bc8b74cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a092c8c-ftf9n" Oct 30 21:45:32.497373 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:32.497348 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50e120fc-8f50-41eb-81f2-71e5bc8b74cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a092c8c-ftf9n\" (UID: \"50e120fc-8f50-41eb-81f2-71e5bc8b74cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a092c8c-ftf9n" Oct 30 21:45:32.497544 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:32.497500 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k2fz\" (UniqueName: \"kubernetes.io/projected/50e120fc-8f50-41eb-81f2-71e5bc8b74cf-kube-api-access-5k2fz\") pod \"maas-api-revoke-sub-5a092c8c-ftf9n\" (UID: \"50e120fc-8f50-41eb-81f2-71e5bc8b74cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a092c8c-ftf9n" Oct 30 21:45:32.498483 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:32.498459 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50e120fc-8f50-41eb-81f2-71e5bc8b74cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a092c8c-ftf9n\" (UID: \"50e120fc-8f50-41eb-81f2-71e5bc8b74cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a092c8c-ftf9n" Oct 30 21:45:32.500330 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:32.500284 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50e120fc-8f50-41eb-81f2-71e5bc8b74cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a092c8c-ftf9n\" (UID: \"50e120fc-8f50-41eb-81f2-71e5bc8b74cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a092c8c-ftf9n" Oct 30 21:45:32.560328 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:32.560293 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k2fz\" (UniqueName: \"kubernetes.io/projected/50e120fc-8f50-41eb-81f2-71e5bc8b74cf-kube-api-access-5k2fz\") pod \"maas-api-revoke-sub-5a092c8c-ftf9n\" (UID: \"50e120fc-8f50-41eb-81f2-71e5bc8b74cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a092c8c-ftf9n" Oct 30 21:45:32.583178 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:32.583136 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ad80cc-cqmzb" Oct 30 21:45:32.598822 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:32.598791 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99cf5ffb-5049-4f6b-aca5-4f6b22c55483-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99cf5ffb-5049-4f6b-aca5-4f6b22c55483-maas-api-openshift-ca\") pod \"99cf5ffb-5049-4f6b-aca5-4f6b22c55483\" (UID: \"99cf5ffb-5049-4f6b-aca5-4f6b22c55483\") " Oct 30 21:45:32.599063 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:32.598852 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99cf5ffb-5049-4f6b-aca5-4f6b22c55483-kube-api-access-lx7wf\" (UniqueName: \"kubernetes.io/projected/99cf5ffb-5049-4f6b-aca5-4f6b22c55483-kube-api-access-lx7wf\") pod \"99cf5ffb-5049-4f6b-aca5-4f6b22c55483\" (UID: \"99cf5ffb-5049-4f6b-aca5-4f6b22c55483\") " Oct 30 21:45:32.599063 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:32.598887 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99cf5ffb-5049-4f6b-aca5-4f6b22c55483-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99cf5ffb-5049-4f6b-aca5-4f6b22c55483-maas-api-xks-ca\") pod \"99cf5ffb-5049-4f6b-aca5-4f6b22c55483\" (UID: \"99cf5ffb-5049-4f6b-aca5-4f6b22c55483\") " Oct 30 21:45:32.599287 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:32.599255 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99cf5ffb-5049-4f6b-aca5-4f6b22c55483-maas-api-openshift-ca" pod "99cf5ffb-5049-4f6b-aca5-4f6b22c55483" (UID: "99cf5ffb-5049-4f6b-aca5-4f6b22c55483"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:32.599816 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:32.599787 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99cf5ffb-5049-4f6b-aca5-4f6b22c55483-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:32.601316 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:32.601277 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99cf5ffb-5049-4f6b-aca5-4f6b22c55483-kube-api-access-lx7wf" pod "99cf5ffb-5049-4f6b-aca5-4f6b22c55483" (UID: "99cf5ffb-5049-4f6b-aca5-4f6b22c55483"). InnerVolumeSpecName "kube-api-access-lx7wf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:32.601840 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:32.601816 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99cf5ffb-5049-4f6b-aca5-4f6b22c55483-maas-api-xks-ca" pod "99cf5ffb-5049-4f6b-aca5-4f6b22c55483" (UID: "99cf5ffb-5049-4f6b-aca5-4f6b22c55483"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:45:32.615677 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:32.615658 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a092c8c-ftf9n" Oct 30 21:45:32.701274 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:32.701213 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx7wf\" (UniqueName: \"kubernetes.io/projected/99cf5ffb-5049-4f6b-aca5-4f6b22c55483-kube-api-access-lx7wf\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:32.701274 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:32.701246 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99cf5ffb-5049-4f6b-aca5-4f6b22c55483-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:33.005097 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:33.005068 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a092c8c-ftf9n"] Oct 30 21:45:33.006450 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:45:33.006409 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50e120fc_8f50_41eb_81f2_71e5bc8b74cf.slice/crio-5db18394c679cc9c24406520251f6f5125d2df605d44cbe98e5c4b0fdece10cb WatchSource:0}: Error finding container 5db18394c679cc9c24406520251f6f5125d2df605d44cbe98e5c4b0fdece10cb: Status 404 returned error can't find the container with id 5db18394c679cc9c24406520251f6f5125d2df605d44cbe98e5c4b0fdece10cb Oct 30 21:45:33.421257 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:33.421222 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a092c8c-ftf9n" event={"ID":"50e120fc-8f50-41eb-81f2-71e5bc8b74cf","Type":"ContainerStarted","Data":"5db18394c679cc9c24406520251f6f5125d2df605d44cbe98e5c4b0fdece10cb"} Oct 30 21:45:33.422632 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:33.422611 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ad80cc-cqmzb" Oct 30 21:45:33.422703 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:33.422631 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ad80cc-cqmzb" event={"ID":"99cf5ffb-5049-4f6b-aca5-4f6b22c55483","Type":"ContainerDied","Data":"005ab48d31061cb1075b732110df963008312e7766cd7a164b2fffc3b11f1577"} Oct 30 21:45:33.422703 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:33.422677 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="005ab48d31061cb1075b732110df963008312e7766cd7a164b2fffc3b11f1577" Oct 30 21:45:34.432639 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:34.432602 5748 generic.go:356] "Generic (PLEG): container finished" podID="50e120fc-8f50-41eb-81f2-71e5bc8b74cf" containerID="576c29020555cc99c86111fdb51059344b8520258c9e8ec35644809b768aba92" exitCode=0 Oct 30 21:45:34.432639 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:34.432640 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a092c8c-ftf9n" event={"ID":"50e120fc-8f50-41eb-81f2-71e5bc8b74cf","Type":"ContainerDied","Data":"576c29020555cc99c86111fdb51059344b8520258c9e8ec35644809b768aba92"} Oct 30 21:45:35.606467 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:35.606427 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a092c8c-ftf9n" Oct 30 21:45:35.640651 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:35.640620 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50e120fc-8f50-41eb-81f2-71e5bc8b74cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50e120fc-8f50-41eb-81f2-71e5bc8b74cf-maas-api-openshift-ca\") pod \"50e120fc-8f50-41eb-81f2-71e5bc8b74cf\" (UID: \"50e120fc-8f50-41eb-81f2-71e5bc8b74cf\") " Oct 30 21:45:35.640803 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:35.640788 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50e120fc-8f50-41eb-81f2-71e5bc8b74cf-kube-api-access-5k2fz\" (UniqueName: \"kubernetes.io/projected/50e120fc-8f50-41eb-81f2-71e5bc8b74cf-kube-api-access-5k2fz\") pod \"50e120fc-8f50-41eb-81f2-71e5bc8b74cf\" (UID: \"50e120fc-8f50-41eb-81f2-71e5bc8b74cf\") " Oct 30 21:45:35.640953 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:35.640935 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50e120fc-8f50-41eb-81f2-71e5bc8b74cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50e120fc-8f50-41eb-81f2-71e5bc8b74cf-maas-api-xks-ca\") pod \"50e120fc-8f50-41eb-81f2-71e5bc8b74cf\" (UID: \"50e120fc-8f50-41eb-81f2-71e5bc8b74cf\") " Oct 30 21:45:35.641047 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:35.641022 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50e120fc-8f50-41eb-81f2-71e5bc8b74cf-maas-api-openshift-ca" pod "50e120fc-8f50-41eb-81f2-71e5bc8b74cf" (UID: "50e120fc-8f50-41eb-81f2-71e5bc8b74cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:35.641492 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:35.641472 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50e120fc-8f50-41eb-81f2-71e5bc8b74cf-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:35.642965 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:35.642940 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50e120fc-8f50-41eb-81f2-71e5bc8b74cf-maas-api-xks-ca" pod "50e120fc-8f50-41eb-81f2-71e5bc8b74cf" (UID: "50e120fc-8f50-41eb-81f2-71e5bc8b74cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:45:35.643056 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:35.642997 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50e120fc-8f50-41eb-81f2-71e5bc8b74cf-kube-api-access-5k2fz" pod "50e120fc-8f50-41eb-81f2-71e5bc8b74cf" (UID: "50e120fc-8f50-41eb-81f2-71e5bc8b74cf"). InnerVolumeSpecName "kube-api-access-5k2fz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:35.743260 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:35.743177 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5k2fz\" (UniqueName: \"kubernetes.io/projected/50e120fc-8f50-41eb-81f2-71e5bc8b74cf-kube-api-access-5k2fz\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:35.743260 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:35.743206 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50e120fc-8f50-41eb-81f2-71e5bc8b74cf-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:36.450858 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:36.450825 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a092c8c-ftf9n" Oct 30 21:45:36.451058 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:36.450860 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a092c8c-ftf9n" event={"ID":"50e120fc-8f50-41eb-81f2-71e5bc8b74cf","Type":"ContainerDied","Data":"5db18394c679cc9c24406520251f6f5125d2df605d44cbe98e5c4b0fdece10cb"} Oct 30 21:45:36.451058 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:36.450927 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5db18394c679cc9c24406520251f6f5125d2df605d44cbe98e5c4b0fdece10cb" Oct 30 21:45:36.605223 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:36.605186 5748 scope.go:122] "RemoveContainer" containerID="41f99d8eac3e86214cc7c9ee2890487405b9742401266e6d314342726ad7a849" Oct 30 21:45:36.615909 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:36.615870 5748 scope.go:122] "RemoveContainer" containerID="1716de9d85339349e8e86dd11cdddf865ac511264472f19054fd5ab75af00296" Oct 30 21:45:36.625807 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:36.625784 5748 scope.go:122] "RemoveContainer" containerID="28e43d94a32926571dbc160ea478ba5200a9ea8a164cf0ec0b1f8c0f7af4ae5f" Oct 30 21:45:36.635670 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:36.635647 5748 scope.go:122] "RemoveContainer" containerID="1093f1c9cb68bf0c843b2f2a223ae0dc36f925393964820ad82bceb5849849ad" Oct 30 21:45:36.645836 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:36.645813 5748 scope.go:122] "RemoveContainer" containerID="22d225f7c37e0858ef6b82f8ac2a5436dab69ac2a03f336291ef8259e768e2a1" Oct 30 21:45:36.661333 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:36.661306 5748 scope.go:122] "RemoveContainer" containerID="77d592c5ea301a2e40fc8ff01bac5fe51567ffccd094ccdfc139f6d37e1bd395" Oct 30 21:45:36.671723 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:36.671704 5748 scope.go:122] "RemoveContainer" containerID="4815dce7efedf569d0f7a2be749484e87c76202a0f35b07c489eddefc4a18a33" Oct 30 21:45:36.681707 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:36.681683 5748 scope.go:122] "RemoveContainer" containerID="22234416d4bdfa4ff363df77eadb28a24a61cb66a8aa6cfd3b77f4bb98203c2c" Oct 30 21:45:36.691771 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:36.691751 5748 scope.go:122] "RemoveContainer" containerID="5c0dd171563e3cdd2b99ad1439106487b1e785aeb9f1a0ce12f3f779ae9c9150" Oct 30 21:45:36.701367 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:36.701347 5748 scope.go:122] "RemoveContainer" containerID="e0f5bac11a634bac11fdc6677dde434b3cb92d58f118a10d1113f383c3309110" Oct 30 21:45:36.710394 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:36.710377 5748 scope.go:122] "RemoveContainer" containerID="d0962ec0ee700ccb488e9c46139a137c75e30b5f0d0fd18add924e376c837873" Oct 30 21:45:36.719488 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:36.719464 5748 scope.go:122] "RemoveContainer" containerID="86f7d7e6a2002c1c763b509faac025d1e9739b75c583f81ab363d3318fceea47" Oct 30 21:45:36.728740 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:36.728719 5748 scope.go:122] "RemoveContainer" containerID="0ac75adabb16aa8c20f930ed700b0f92a280a61275033f0f7a8b7168f92fb66f" Oct 30 21:45:39.636581 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:39.636543 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf97b89a-qdkwh"] Oct 30 21:45:39.637143 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:39.637128 5748 cpu_manager.go:414] "containerMap: removing container" podUID="99cf5ffb-5049-4f6b-aca5-4f6b22c55483" containerName="revoke-keys" Oct 30 21:45:39.637194 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:39.637143 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99cf5ffb-5049-4f6b-aca5-4f6b22c55483" containerName="revoke-keys" Oct 30 21:45:39.637194 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:39.637157 5748 cpu_manager.go:414] "containerMap: removing container" podUID="50e120fc-8f50-41eb-81f2-71e5bc8b74cf" containerName="revoke-keys" Oct 30 21:45:39.637194 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:39.637162 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50e120fc-8f50-41eb-81f2-71e5bc8b74cf" containerName="revoke-keys" Oct 30 21:45:39.637290 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:39.637286 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="99cf5ffb-5049-4f6b-aca5-4f6b22c55483" containerName="revoke-keys" Oct 30 21:45:39.637324 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:39.637296 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="50e120fc-8f50-41eb-81f2-71e5bc8b74cf" containerName="revoke-keys" Oct 30 21:45:39.644578 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:39.644558 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf97b89a-qdkwh" Oct 30 21:45:39.699822 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:39.699786 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bfgl\" (UniqueName: \"kubernetes.io/projected/ddcc2107-c0c9-4ddd-ba34-66d000b6d204-kube-api-access-4bfgl\") pod \"maas-api-revoke-sub-bf97b89a-qdkwh\" (UID: \"ddcc2107-c0c9-4ddd-ba34-66d000b6d204\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf97b89a-qdkwh" Oct 30 21:45:39.700052 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:39.699832 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddcc2107-c0c9-4ddd-ba34-66d000b6d204-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf97b89a-qdkwh\" (UID: \"ddcc2107-c0c9-4ddd-ba34-66d000b6d204\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf97b89a-qdkwh" Oct 30 21:45:39.700052 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:39.699975 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddcc2107-c0c9-4ddd-ba34-66d000b6d204-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf97b89a-qdkwh\" (UID: \"ddcc2107-c0c9-4ddd-ba34-66d000b6d204\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf97b89a-qdkwh" Oct 30 21:45:39.734591 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:39.734556 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:45:39.734776 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:39.734653 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:45:39.734776 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:39.734706 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 21:45:39.734776 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:39.734722 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:45:39.741375 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:39.741341 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf97b89a-qdkwh"] Oct 30 21:45:39.801801 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:39.801766 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bfgl\" (UniqueName: \"kubernetes.io/projected/ddcc2107-c0c9-4ddd-ba34-66d000b6d204-kube-api-access-4bfgl\") pod \"maas-api-revoke-sub-bf97b89a-qdkwh\" (UID: \"ddcc2107-c0c9-4ddd-ba34-66d000b6d204\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf97b89a-qdkwh" Oct 30 21:45:39.801989 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:39.801814 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddcc2107-c0c9-4ddd-ba34-66d000b6d204-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf97b89a-qdkwh\" (UID: \"ddcc2107-c0c9-4ddd-ba34-66d000b6d204\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf97b89a-qdkwh" Oct 30 21:45:39.801989 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:39.801912 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddcc2107-c0c9-4ddd-ba34-66d000b6d204-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf97b89a-qdkwh\" (UID: \"ddcc2107-c0c9-4ddd-ba34-66d000b6d204\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf97b89a-qdkwh" Oct 30 21:45:39.802641 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:39.802619 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddcc2107-c0c9-4ddd-ba34-66d000b6d204-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf97b89a-qdkwh\" (UID: \"ddcc2107-c0c9-4ddd-ba34-66d000b6d204\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf97b89a-qdkwh" Oct 30 21:45:39.804271 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:39.804247 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddcc2107-c0c9-4ddd-ba34-66d000b6d204-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf97b89a-qdkwh\" (UID: \"ddcc2107-c0c9-4ddd-ba34-66d000b6d204\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf97b89a-qdkwh" Oct 30 21:45:39.863680 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:39.863649 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bfgl\" (UniqueName: \"kubernetes.io/projected/ddcc2107-c0c9-4ddd-ba34-66d000b6d204-kube-api-access-4bfgl\") pod \"maas-api-revoke-sub-bf97b89a-qdkwh\" (UID: \"ddcc2107-c0c9-4ddd-ba34-66d000b6d204\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf97b89a-qdkwh" Oct 30 21:45:39.958361 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:39.958327 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf97b89a-qdkwh" Oct 30 21:45:40.369182 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:40.369157 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf97b89a-qdkwh"] Oct 30 21:45:40.369444 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:45:40.369423 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddcc2107_c0c9_4ddd_ba34_66d000b6d204.slice/crio-b8d1f293857fc9d7c5123af5e782c3ebdbf0f71b337cb30dbcc168a8a4019010 WatchSource:0}: Error finding container b8d1f293857fc9d7c5123af5e782c3ebdbf0f71b337cb30dbcc168a8a4019010: Status 404 returned error can't find the container with id b8d1f293857fc9d7c5123af5e782c3ebdbf0f71b337cb30dbcc168a8a4019010 Oct 30 21:45:40.482615 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:40.482576 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf97b89a-qdkwh" event={"ID":"ddcc2107-c0c9-4ddd-ba34-66d000b6d204","Type":"ContainerStarted","Data":"b8d1f293857fc9d7c5123af5e782c3ebdbf0f71b337cb30dbcc168a8a4019010"} Oct 30 21:45:41.491663 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:41.491576 5748 generic.go:356] "Generic (PLEG): container finished" podID="ddcc2107-c0c9-4ddd-ba34-66d000b6d204" containerID="1b21f4b182253975068da96238dcb8f64e9bc59bd1966122e169d671ab1397e3" exitCode=0 Oct 30 21:45:41.491663 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:41.491622 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf97b89a-qdkwh" event={"ID":"ddcc2107-c0c9-4ddd-ba34-66d000b6d204","Type":"ContainerDied","Data":"1b21f4b182253975068da96238dcb8f64e9bc59bd1966122e169d671ab1397e3"} Oct 30 21:45:42.661095 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:42.661066 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf97b89a-qdkwh" Oct 30 21:45:42.742525 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:42.742485 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddcc2107-c0c9-4ddd-ba34-66d000b6d204-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddcc2107-c0c9-4ddd-ba34-66d000b6d204-maas-api-xks-ca\") pod \"ddcc2107-c0c9-4ddd-ba34-66d000b6d204\" (UID: \"ddcc2107-c0c9-4ddd-ba34-66d000b6d204\") " Oct 30 21:45:42.742720 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:42.742614 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddcc2107-c0c9-4ddd-ba34-66d000b6d204-kube-api-access-4bfgl\" (UniqueName: \"kubernetes.io/projected/ddcc2107-c0c9-4ddd-ba34-66d000b6d204-kube-api-access-4bfgl\") pod \"ddcc2107-c0c9-4ddd-ba34-66d000b6d204\" (UID: \"ddcc2107-c0c9-4ddd-ba34-66d000b6d204\") " Oct 30 21:45:42.742812 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:42.742798 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddcc2107-c0c9-4ddd-ba34-66d000b6d204-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddcc2107-c0c9-4ddd-ba34-66d000b6d204-maas-api-openshift-ca\") pod \"ddcc2107-c0c9-4ddd-ba34-66d000b6d204\" (UID: \"ddcc2107-c0c9-4ddd-ba34-66d000b6d204\") " Oct 30 21:45:42.743132 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:42.743109 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddcc2107-c0c9-4ddd-ba34-66d000b6d204-maas-api-openshift-ca" pod "ddcc2107-c0c9-4ddd-ba34-66d000b6d204" (UID: "ddcc2107-c0c9-4ddd-ba34-66d000b6d204"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:42.743419 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:42.743400 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddcc2107-c0c9-4ddd-ba34-66d000b6d204-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:42.744602 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:42.744582 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddcc2107-c0c9-4ddd-ba34-66d000b6d204-maas-api-xks-ca" pod "ddcc2107-c0c9-4ddd-ba34-66d000b6d204" (UID: "ddcc2107-c0c9-4ddd-ba34-66d000b6d204"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:45:42.744681 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:42.744656 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddcc2107-c0c9-4ddd-ba34-66d000b6d204-kube-api-access-4bfgl" pod "ddcc2107-c0c9-4ddd-ba34-66d000b6d204" (UID: "ddcc2107-c0c9-4ddd-ba34-66d000b6d204"). InnerVolumeSpecName "kube-api-access-4bfgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:42.844827 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:42.844792 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddcc2107-c0c9-4ddd-ba34-66d000b6d204-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:42.844827 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:42.844822 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bfgl\" (UniqueName: \"kubernetes.io/projected/ddcc2107-c0c9-4ddd-ba34-66d000b6d204-kube-api-access-4bfgl\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:43.507884 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:43.507853 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf97b89a-qdkwh" Oct 30 21:45:43.507884 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:43.507891 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf97b89a-qdkwh" event={"ID":"ddcc2107-c0c9-4ddd-ba34-66d000b6d204","Type":"ContainerDied","Data":"b8d1f293857fc9d7c5123af5e782c3ebdbf0f71b337cb30dbcc168a8a4019010"} Oct 30 21:45:43.508136 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:43.507975 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8d1f293857fc9d7c5123af5e782c3ebdbf0f71b337cb30dbcc168a8a4019010" Oct 30 21:45:47.409923 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:47.396794 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h" podUID="0a0b0425-2959-4690-9847-38852e167230" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Oct 30 21:45:47.467941 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:47.444973 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ea374/distinct-2-w1-6ea374-kserve-68fb49889-hpg2k" podUID="ab533734-e0d8-4acb-854d-12851a87bc58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:47.524253 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:47.507860 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ea374/simulator-w1-6ea374-kserve-775cfdf98b-p5q25" podUID="b0d53207-65f3-4d43-baba-66bc4c366885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 21:45:47.839979 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:47.829589 5748 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-d68156/distinct-w3-d68156-kserve-565894fdb4-85t7z" podUID="13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:47.886922 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:47.886001 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d68156/distinct-w3-d68156-kserve-565894fdb4-85t7z" podUID="13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 21:45:47.886922 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:47.886377 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-69e4b2/simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj" podUID="698fc0f2-64ce-4e7e-b482-8ebd9f33f84d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:47.936923 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:47.927399 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c7d477/distinct-w5-c7d477-kserve-554d7f974d-6zw6s" podUID="09d39cc0-b68d-48fd-918c-7d2518f384aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:47.936923 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:47.929264 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1de75a/simulator-w4-1de75a-kserve-6887dd4b46-kc5c2" podUID="926fa847-0c3c-43e0-b378-04c82433cd98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Oct 30 21:45:48.008936 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:48.007108 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91874e/distinct-2-w2-91874e-kserve-69fcf64f44-qm96r" podUID="0cb16f33-b208-4c11-bb12-322829bd51e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:48.048925 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:48.046697 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d68156/simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk" podUID="db285e9c-ca87-43fa-8f06-69e271e5c2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:48.052934 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:48.051062 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91874e/distinct-w2-91874e-kserve-58d6bdfc75-bdp5z" podUID="cb39b415-f406-47bd-b591-be7e09240f5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:48.061923 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:48.059969 5748 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-91874e/distinct-w2-91874e-kserve-58d6bdfc75-bdp5z" podUID="cb39b415-f406-47bd-b591-be7e09240f5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:48.061923 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:48.060060 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-691598/simulator-w0-691598-kserve-76fb5d5bc7-l5gqk" podUID="4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 30 21:45:48.253923 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:48.252989 5748 patch_prober.go:29] interesting pod/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:45:48.253923 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:48.253072 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9" podUID="ab385653-5874-4072-a986-201ae88230cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:48.377037 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:48.376995 5748 patch_prober.go:29] interesting pod/e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:45:48.378099 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:48.378065 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s" podUID="93b2f11c-31c6-49c6-a033-c79eaaa4f7c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:48.452923 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:48.451979 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h" podUID="0a0b0425-2959-4690-9847-38852e167230" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Oct 30 21:45:48.452923 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:48.452424 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h" Oct 30 21:45:48.472938 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:48.469998 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-69e4b2/premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b" podUID="3ed43145-33bd-4c11-bb29-dc852511f6d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:48.512583 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:48.512484 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ea374/distinct-2-w1-6ea374-kserve-68fb49889-hpg2k" podUID="ab533734-e0d8-4acb-854d-12851a87bc58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:48.515688 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:48.513303 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6ea374/distinct-2-w1-6ea374-kserve-68fb49889-hpg2k" Oct 30 21:45:48.515688 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:48.512812 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ea374/simulator-w1-6ea374-kserve-775cfdf98b-p5q25" podUID="b0d53207-65f3-4d43-baba-66bc4c366885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:48.515688 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:48.514542 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6ea374/simulator-w1-6ea374-kserve-775cfdf98b-p5q25" Oct 30 21:45:48.946922 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:48.892112 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-69e4b2/simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj" podUID="698fc0f2-64ce-4e7e-b482-8ebd9f33f84d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:49.234470 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:48.925291 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d68156/distinct-w3-d68156-kserve-565894fdb4-85t7z" podUID="13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:49.423920 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:48.948240 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1de75a/simulator-w4-1de75a-kserve-6887dd4b46-kc5c2" podUID="926fa847-0c3c-43e0-b378-04c82433cd98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:49.423920 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:48.948275 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c7d477/distinct-w5-c7d477-kserve-554d7f974d-6zw6s" podUID="09d39cc0-b68d-48fd-918c-7d2518f384aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Oct 30 21:45:49.423920 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:48.949278 5748 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h" podUID="0a0b0425-2959-4690-9847-38852e167230" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:49.423920 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:49.016153 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91874e/distinct-2-w2-91874e-kserve-69fcf64f44-qm96r" podUID="0cb16f33-b208-4c11-bb12-322829bd51e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Oct 30 21:45:49.423920 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:49.120215 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91874e/distinct-w2-91874e-kserve-58d6bdfc75-bdp5z" podUID="cb39b415-f406-47bd-b591-be7e09240f5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:49.423920 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:49.120260 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d68156/simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk" podUID="db285e9c-ca87-43fa-8f06-69e271e5c2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 30 21:45:49.423920 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:49.136950 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-691598/simulator-w0-691598-kserve-76fb5d5bc7-l5gqk" podUID="4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:49.423920 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:49.238855 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-69e4b2/simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj" Oct 30 21:45:49.423920 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:49.421762 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d68156/distinct-w3-d68156-kserve-565894fdb4-85t7z" Oct 30 21:45:49.423920 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:49.421859 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1de75a/simulator-w4-1de75a-kserve-6887dd4b46-kc5c2" Oct 30 21:45:49.423920 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:49.421922 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c7d477/distinct-w5-c7d477-kserve-554d7f974d-6zw6s" Oct 30 21:45:49.423920 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:49.421959 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-91874e/distinct-2-w2-91874e-kserve-69fcf64f44-qm96r" Oct 30 21:45:49.423920 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:49.421994 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-91874e/distinct-w2-91874e-kserve-58d6bdfc75-bdp5z" Oct 30 21:45:49.423920 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:49.422025 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d68156/simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk" Oct 30 21:45:49.423920 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:49.422058 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-691598/simulator-w0-691598-kserve-76fb5d5bc7-l5gqk" Oct 30 21:45:49.423920 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:49.382134 5748 patch_prober.go:29] interesting pod/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:45:49.423920 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:49.422943 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9" podUID="0018c42f-45f5-4417-8b83-6d0cd9bd3c02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:49.651017 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:49.571016 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ea374/distinct-2-w1-6ea374-kserve-68fb49889-hpg2k" podUID="ab533734-e0d8-4acb-854d-12851a87bc58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 30 21:45:49.688964 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:49.652702 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h" podUID="0a0b0425-2959-4690-9847-38852e167230" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:49.724922 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:49.706158 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ea374/simulator-w1-6ea374-kserve-775cfdf98b-p5q25" podUID="b0d53207-65f3-4d43-baba-66bc4c366885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:49.769098 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:49.769041 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-69e4b2/premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b" podUID="3ed43145-33bd-4c11-bb29-dc852511f6d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:49.773264 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:49.773235 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-69e4b2/premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b" Oct 30 21:45:50.185941 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:50.167543 5748 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-d68156/simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk" podUID="db285e9c-ca87-43fa-8f06-69e271e5c2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:50.266050 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:50.266002 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-69e4b2/simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj" podUID="698fc0f2-64ce-4e7e-b482-8ebd9f33f84d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 30 21:45:50.518979 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:50.463096 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91874e/distinct-2-w2-91874e-kserve-69fcf64f44-qm96r" podUID="0cb16f33-b208-4c11-bb12-322829bd51e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Oct 30 21:45:50.518979 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:50.463465 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d68156/distinct-w3-d68156-kserve-565894fdb4-85t7z" podUID="13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 21:45:50.518979 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:50.463708 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1de75a/simulator-w4-1de75a-kserve-6887dd4b46-kc5c2" podUID="926fa847-0c3c-43e0-b378-04c82433cd98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Oct 30 21:45:50.518979 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:50.463945 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c7d477/distinct-w5-c7d477-kserve-554d7f974d-6zw6s" podUID="09d39cc0-b68d-48fd-918c-7d2518f384aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Oct 30 21:45:50.612923 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:50.541511 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d68156/simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk" podUID="db285e9c-ca87-43fa-8f06-69e271e5c2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 30 21:45:50.637924 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:50.545585 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91874e/distinct-w2-91874e-kserve-58d6bdfc75-bdp5z" podUID="cb39b415-f406-47bd-b591-be7e09240f5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 30 21:45:50.638706 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:50.545642 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-691598/simulator-w0-691598-kserve-76fb5d5bc7-l5gqk" podUID="4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 30 21:45:50.639151 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:50.546881 5748 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6ea374/distinct-2-w1-6ea374-kserve-68fb49889-hpg2k" podUID="ab533734-e0d8-4acb-854d-12851a87bc58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:50.646921 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:50.578438 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ea374/distinct-2-w1-6ea374-kserve-68fb49889-hpg2k" podUID="ab533734-e0d8-4acb-854d-12851a87bc58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:50.718923 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:50.705931 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bh2l5" podUID="62a4e725-cffd-4326-b072-506423d26fc5" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.23:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:50.719568 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:50.719526 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h" podUID="0a0b0425-2959-4690-9847-38852e167230" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Oct 30 21:45:50.803232 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:50.774014 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-69e4b2/premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b" podUID="3ed43145-33bd-4c11-bb29-dc852511f6d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:50.987929 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:50.981386 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6ea374/simulator-w1-6ea374-kserve-775cfdf98b-p5q25" Oct 30 21:45:51.045918 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:51.041940 5748 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-91874e/distinct-2-w2-91874e-kserve-69fcf64f44-qm96r" podUID="0cb16f33-b208-4c11-bb12-322829bd51e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:51.241129 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:51.198855 5748 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-691598/simulator-w0-691598-kserve-76fb5d5bc7-l5gqk" podUID="4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:51.242040 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:51.242006 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6ea374/distinct-2-w1-6ea374-kserve-68fb49889-hpg2k" Oct 30 21:45:51.349404 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:51.349353 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-69e4b2/simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj" podUID="698fc0f2-64ce-4e7e-b482-8ebd9f33f84d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 30 21:45:51.472984 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:51.472940 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c7d477/distinct-w5-c7d477-kserve-554d7f974d-6zw6s" podUID="09d39cc0-b68d-48fd-918c-7d2518f384aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:51.474112 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:51.474070 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91874e/distinct-2-w2-91874e-kserve-69fcf64f44-qm96r" podUID="0cb16f33-b208-4c11-bb12-322829bd51e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Oct 30 21:45:51.475050 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:51.475024 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d68156/distinct-w3-d68156-kserve-565894fdb4-85t7z" podUID="13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 21:45:51.475857 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:51.475820 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1de75a/simulator-w4-1de75a-kserve-6887dd4b46-kc5c2" podUID="926fa847-0c3c-43e0-b378-04c82433cd98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:51.631728 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:51.631624 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d68156/simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk" podUID="db285e9c-ca87-43fa-8f06-69e271e5c2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 30 21:45:51.694055 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:51.694008 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ea374/simulator-w1-6ea374-kserve-775cfdf98b-p5q25" podUID="b0d53207-65f3-4d43-baba-66bc4c366885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 21:45:51.712912 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:51.700970 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91874e/distinct-w2-91874e-kserve-58d6bdfc75-bdp5z" podUID="cb39b415-f406-47bd-b591-be7e09240f5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 30 21:45:51.740857 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:51.740805 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h" podUID="0a0b0425-2959-4690-9847-38852e167230" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Oct 30 21:45:51.798608 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:51.798549 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-691598/simulator-w0-691598-kserve-76fb5d5bc7-l5gqk" podUID="4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:51.835202 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:51.835143 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-69e4b2/premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b" podUID="3ed43145-33bd-4c11-bb29-dc852511f6d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:52.094285 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:52.092457 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-691598/simulator-w0-691598-kserve-76fb5d5bc7-l5gqk" Oct 30 21:45:52.147627 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:52.145447 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h" Oct 30 21:45:52.250872 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:52.250830 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1de75a/simulator-w4-1de75a-kserve-6887dd4b46-kc5c2" Oct 30 21:45:52.252140 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:52.252116 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d68156/simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk" Oct 30 21:45:52.318257 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:52.317186 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-69e4b2/premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b" Oct 30 21:45:52.358132 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:52.358005 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-69e4b2/simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj" podUID="698fc0f2-64ce-4e7e-b482-8ebd9f33f84d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:52.522259 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:52.499363 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c7d477/distinct-w5-c7d477-kserve-554d7f974d-6zw6s" podUID="09d39cc0-b68d-48fd-918c-7d2518f384aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:52.522259 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:52.500107 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91874e/distinct-2-w2-91874e-kserve-69fcf64f44-qm96r" podUID="0cb16f33-b208-4c11-bb12-322829bd51e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Oct 30 21:45:52.522259 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:52.500341 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d68156/distinct-w3-d68156-kserve-565894fdb4-85t7z" podUID="13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 21:45:52.710669 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:52.710637 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-91874e/distinct-w2-91874e-kserve-58d6bdfc75-bdp5z" Oct 30 21:45:52.710943 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:52.710928 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-91874e/distinct-2-w2-91874e-kserve-69fcf64f44-qm96r" Oct 30 21:45:52.711091 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:52.711079 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d68156/distinct-w3-d68156-kserve-565894fdb4-85t7z" Oct 30 21:45:52.711215 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:52.711205 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-69e4b2/simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj" Oct 30 21:45:52.732923 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:52.730043 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c7d477/distinct-w5-c7d477-kserve-554d7f974d-6zw6s" Oct 30 21:45:59.347221 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:59.347179 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-948a35c6-pbgz4"] Oct 30 21:45:59.347791 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:59.347774 5748 cpu_manager.go:414] "containerMap: removing container" podUID="ddcc2107-c0c9-4ddd-ba34-66d000b6d204" containerName="revoke-keys" Oct 30 21:45:59.347791 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:59.347790 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddcc2107-c0c9-4ddd-ba34-66d000b6d204" containerName="revoke-keys" Oct 30 21:45:59.347965 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:59.347909 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddcc2107-c0c9-4ddd-ba34-66d000b6d204" containerName="revoke-keys" Oct 30 21:45:59.353177 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:59.353154 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-948a35c6-pbgz4" Oct 30 21:45:59.426932 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:59.426866 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-948a35c6-pbgz4"] Oct 30 21:45:59.455305 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:59.455267 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:45:59.455305 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:59.455288 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:45:59.455305 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:59.455265 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:45:59.456083 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:59.456064 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 21:45:59.476295 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:59.476253 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fscjl\" (UniqueName: \"kubernetes.io/projected/58893d7b-61e4-4e3a-b7fe-1f0d743d9fa6-kube-api-access-fscjl\") pod \"maas-api-revoke-sub-948a35c6-pbgz4\" (UID: \"58893d7b-61e4-4e3a-b7fe-1f0d743d9fa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-948a35c6-pbgz4" Oct 30 21:45:59.476710 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:59.476666 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58893d7b-61e4-4e3a-b7fe-1f0d743d9fa6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-948a35c6-pbgz4\" (UID: \"58893d7b-61e4-4e3a-b7fe-1f0d743d9fa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-948a35c6-pbgz4" Oct 30 21:45:59.476836 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:59.476722 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58893d7b-61e4-4e3a-b7fe-1f0d743d9fa6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-948a35c6-pbgz4\" (UID: \"58893d7b-61e4-4e3a-b7fe-1f0d743d9fa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-948a35c6-pbgz4" Oct 30 21:45:59.578911 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:59.578865 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58893d7b-61e4-4e3a-b7fe-1f0d743d9fa6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-948a35c6-pbgz4\" (UID: \"58893d7b-61e4-4e3a-b7fe-1f0d743d9fa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-948a35c6-pbgz4" Oct 30 21:45:59.579130 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:59.578945 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58893d7b-61e4-4e3a-b7fe-1f0d743d9fa6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-948a35c6-pbgz4\" (UID: \"58893d7b-61e4-4e3a-b7fe-1f0d743d9fa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-948a35c6-pbgz4" Oct 30 21:45:59.579224 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:59.579134 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fscjl\" (UniqueName: \"kubernetes.io/projected/58893d7b-61e4-4e3a-b7fe-1f0d743d9fa6-kube-api-access-fscjl\") pod \"maas-api-revoke-sub-948a35c6-pbgz4\" (UID: \"58893d7b-61e4-4e3a-b7fe-1f0d743d9fa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-948a35c6-pbgz4" Oct 30 21:45:59.579697 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:59.579668 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58893d7b-61e4-4e3a-b7fe-1f0d743d9fa6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-948a35c6-pbgz4\" (UID: \"58893d7b-61e4-4e3a-b7fe-1f0d743d9fa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-948a35c6-pbgz4" Oct 30 21:45:59.581803 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:59.581776 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58893d7b-61e4-4e3a-b7fe-1f0d743d9fa6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-948a35c6-pbgz4\" (UID: \"58893d7b-61e4-4e3a-b7fe-1f0d743d9fa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-948a35c6-pbgz4" Oct 30 21:45:59.642087 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:59.642054 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fscjl\" (UniqueName: \"kubernetes.io/projected/58893d7b-61e4-4e3a-b7fe-1f0d743d9fa6-kube-api-access-fscjl\") pod \"maas-api-revoke-sub-948a35c6-pbgz4\" (UID: \"58893d7b-61e4-4e3a-b7fe-1f0d743d9fa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-948a35c6-pbgz4" Oct 30 21:45:59.673531 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:45:59.673500 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-948a35c6-pbgz4" Oct 30 21:46:00.148656 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:46:00.148579 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:46:00.155162 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:00.155134 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-948a35c6-pbgz4"] Oct 30 21:46:00.155692 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:46:00.155664 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58893d7b_61e4_4e3a_b7fe_1f0d743d9fa6.slice/crio-da29efec2e11da3d84c9eb5b6bc398fd76f3f2ac82fa4d720611532643964934 WatchSource:0}: Error finding container da29efec2e11da3d84c9eb5b6bc398fd76f3f2ac82fa4d720611532643964934: Status 404 returned error can't find the container with id da29efec2e11da3d84c9eb5b6bc398fd76f3f2ac82fa4d720611532643964934 Oct 30 21:46:01.023946 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:01.023877 5748 generic.go:356] "Generic (PLEG): container finished" podID="58893d7b-61e4-4e3a-b7fe-1f0d743d9fa6" containerID="16170affbad8d56e433d4ec5e75619c900e8f104d34da3d1668ae5ec5894aad8" exitCode=0 Oct 30 21:46:01.023946 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:01.023939 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-948a35c6-pbgz4" event={"ID":"58893d7b-61e4-4e3a-b7fe-1f0d743d9fa6","Type":"ContainerDied","Data":"16170affbad8d56e433d4ec5e75619c900e8f104d34da3d1668ae5ec5894aad8"} Oct 30 21:46:01.024367 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:01.024004 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-948a35c6-pbgz4" event={"ID":"58893d7b-61e4-4e3a-b7fe-1f0d743d9fa6","Type":"ContainerStarted","Data":"da29efec2e11da3d84c9eb5b6bc398fd76f3f2ac82fa4d720611532643964934"} Oct 30 21:46:02.206100 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:02.206071 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-948a35c6-pbgz4" Oct 30 21:46:02.317930 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:02.317831 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58893d7b-61e4-4e3a-b7fe-1f0d743d9fa6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58893d7b-61e4-4e3a-b7fe-1f0d743d9fa6-maas-api-xks-ca\") pod \"58893d7b-61e4-4e3a-b7fe-1f0d743d9fa6\" (UID: \"58893d7b-61e4-4e3a-b7fe-1f0d743d9fa6\") " Oct 30 21:46:02.318100 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:02.317942 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58893d7b-61e4-4e3a-b7fe-1f0d743d9fa6-kube-api-access-fscjl\" (UniqueName: \"kubernetes.io/projected/58893d7b-61e4-4e3a-b7fe-1f0d743d9fa6-kube-api-access-fscjl\") pod \"58893d7b-61e4-4e3a-b7fe-1f0d743d9fa6\" (UID: \"58893d7b-61e4-4e3a-b7fe-1f0d743d9fa6\") " Oct 30 21:46:02.318100 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:02.317987 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58893d7b-61e4-4e3a-b7fe-1f0d743d9fa6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58893d7b-61e4-4e3a-b7fe-1f0d743d9fa6-maas-api-openshift-ca\") pod \"58893d7b-61e4-4e3a-b7fe-1f0d743d9fa6\" (UID: \"58893d7b-61e4-4e3a-b7fe-1f0d743d9fa6\") " Oct 30 21:46:02.318393 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:02.318364 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58893d7b-61e4-4e3a-b7fe-1f0d743d9fa6-maas-api-openshift-ca" pod "58893d7b-61e4-4e3a-b7fe-1f0d743d9fa6" (UID: "58893d7b-61e4-4e3a-b7fe-1f0d743d9fa6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:02.318718 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:02.318695 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58893d7b-61e4-4e3a-b7fe-1f0d743d9fa6-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:02.320195 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:02.320171 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58893d7b-61e4-4e3a-b7fe-1f0d743d9fa6-maas-api-xks-ca" pod "58893d7b-61e4-4e3a-b7fe-1f0d743d9fa6" (UID: "58893d7b-61e4-4e3a-b7fe-1f0d743d9fa6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:46:02.320278 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:02.320258 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58893d7b-61e4-4e3a-b7fe-1f0d743d9fa6-kube-api-access-fscjl" pod "58893d7b-61e4-4e3a-b7fe-1f0d743d9fa6" (UID: "58893d7b-61e4-4e3a-b7fe-1f0d743d9fa6"). InnerVolumeSpecName "kube-api-access-fscjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:02.420170 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:02.420134 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58893d7b-61e4-4e3a-b7fe-1f0d743d9fa6-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:02.420170 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:02.420174 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fscjl\" (UniqueName: \"kubernetes.io/projected/58893d7b-61e4-4e3a-b7fe-1f0d743d9fa6-kube-api-access-fscjl\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:03.041013 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:03.040972 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-948a35c6-pbgz4" event={"ID":"58893d7b-61e4-4e3a-b7fe-1f0d743d9fa6","Type":"ContainerDied","Data":"da29efec2e11da3d84c9eb5b6bc398fd76f3f2ac82fa4d720611532643964934"} Oct 30 21:46:03.041013 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:03.041010 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-948a35c6-pbgz4" Oct 30 21:46:03.041218 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:03.041032 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da29efec2e11da3d84c9eb5b6bc398fd76f3f2ac82fa4d720611532643964934" Oct 30 21:46:07.453047 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.453011 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr"] Oct 30 21:46:07.453637 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.453622 5748 cpu_manager.go:414] "containerMap: removing container" podUID="58893d7b-61e4-4e3a-b7fe-1f0d743d9fa6" containerName="revoke-keys" Oct 30 21:46:07.453700 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.453638 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58893d7b-61e4-4e3a-b7fe-1f0d743d9fa6" containerName="revoke-keys" Oct 30 21:46:07.453762 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.453753 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="58893d7b-61e4-4e3a-b7fe-1f0d743d9fa6" containerName="revoke-keys" Oct 30 21:46:07.457716 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.457691 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr" Oct 30 21:46:07.504080 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.504035 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c"] Oct 30 21:46:07.508225 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.508199 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-djvfd"] Oct 30 21:46:07.508393 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.508330 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c" Oct 30 21:46:07.512631 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.512609 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-djvfd" Oct 30 21:46:07.557595 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.557556 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr"] Oct 30 21:46:07.612483 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.612446 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c"] Oct 30 21:46:07.616724 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.616698 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-djvfd"] Oct 30 21:46:07.623805 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.623779 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38375f9a-62ae-4a49-ab8d-f3074b9ac694-istio-data\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr\" (UID: \"38375f9a-62ae-4a49-ab8d-f3074b9ac694\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr" Oct 30 21:46:07.623961 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.623812 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/325b45b9-2c8e-47a5-9559-7917be8270dd-istio-token\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-djvfd\" (UID: \"325b45b9-2c8e-47a5-9559-7917be8270dd\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-djvfd" Oct 30 21:46:07.623961 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.623848 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38375f9a-62ae-4a49-ab8d-f3074b9ac694-istiod-ca-cert\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr\" (UID: \"38375f9a-62ae-4a49-ab8d-f3074b9ac694\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr" Oct 30 21:46:07.623961 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.623943 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/325b45b9-2c8e-47a5-9559-7917be8270dd-workload-certs\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-djvfd\" (UID: \"325b45b9-2c8e-47a5-9559-7917be8270dd\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-djvfd" Oct 30 21:46:07.624112 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.624029 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f74b9971-1030-4e52-8f27-6139e51d8a75-istio-token\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c\" (UID: \"f74b9971-1030-4e52-8f27-6139e51d8a75\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c" Oct 30 21:46:07.624161 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.624130 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38375f9a-62ae-4a49-ab8d-f3074b9ac694-workload-socket\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr\" (UID: \"38375f9a-62ae-4a49-ab8d-f3074b9ac694\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr" Oct 30 21:46:07.624219 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.624163 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/325b45b9-2c8e-47a5-9559-7917be8270dd-istio-data\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-djvfd\" (UID: \"325b45b9-2c8e-47a5-9559-7917be8270dd\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-djvfd" Oct 30 21:46:07.624273 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.624218 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38375f9a-62ae-4a49-ab8d-f3074b9ac694-credential-socket\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr\" (UID: \"38375f9a-62ae-4a49-ab8d-f3074b9ac694\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr" Oct 30 21:46:07.624273 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.624251 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38375f9a-62ae-4a49-ab8d-f3074b9ac694-istio-envoy\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr\" (UID: \"38375f9a-62ae-4a49-ab8d-f3074b9ac694\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr" Oct 30 21:46:07.624456 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.624431 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f74b9971-1030-4e52-8f27-6139e51d8a75-istio-podinfo\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c\" (UID: \"f74b9971-1030-4e52-8f27-6139e51d8a75\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c" Oct 30 21:46:07.624522 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.624484 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38375f9a-62ae-4a49-ab8d-f3074b9ac694-istio-podinfo\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr\" (UID: \"38375f9a-62ae-4a49-ab8d-f3074b9ac694\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr" Oct 30 21:46:07.624588 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.624571 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f74b9971-1030-4e52-8f27-6139e51d8a75-istio-envoy\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c\" (UID: \"f74b9971-1030-4e52-8f27-6139e51d8a75\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c" Oct 30 21:46:07.624659 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.624621 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38375f9a-62ae-4a49-ab8d-f3074b9ac694-istio-token\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr\" (UID: \"38375f9a-62ae-4a49-ab8d-f3074b9ac694\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr" Oct 30 21:46:07.624744 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.624723 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/325b45b9-2c8e-47a5-9559-7917be8270dd-credential-socket\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-djvfd\" (UID: \"325b45b9-2c8e-47a5-9559-7917be8270dd\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-djvfd" Oct 30 21:46:07.624809 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.624795 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/325b45b9-2c8e-47a5-9559-7917be8270dd-istiod-ca-cert\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-djvfd\" (UID: \"325b45b9-2c8e-47a5-9559-7917be8270dd\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-djvfd" Oct 30 21:46:07.624872 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.624827 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f74b9971-1030-4e52-8f27-6139e51d8a75-istio-data\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c\" (UID: \"f74b9971-1030-4e52-8f27-6139e51d8a75\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c" Oct 30 21:46:07.624872 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.624853 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp7cz\" (UniqueName: \"kubernetes.io/projected/f74b9971-1030-4e52-8f27-6139e51d8a75-kube-api-access-mp7cz\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c\" (UID: \"f74b9971-1030-4e52-8f27-6139e51d8a75\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c" Oct 30 21:46:07.625002 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.624884 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f74b9971-1030-4e52-8f27-6139e51d8a75-istiod-ca-cert\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c\" (UID: \"f74b9971-1030-4e52-8f27-6139e51d8a75\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c" Oct 30 21:46:07.625002 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.624940 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/325b45b9-2c8e-47a5-9559-7917be8270dd-workload-socket\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-djvfd\" (UID: \"325b45b9-2c8e-47a5-9559-7917be8270dd\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-djvfd" Oct 30 21:46:07.625002 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.624970 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcfgb\" (UniqueName: \"kubernetes.io/projected/325b45b9-2c8e-47a5-9559-7917be8270dd-kube-api-access-gcfgb\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-djvfd\" (UID: \"325b45b9-2c8e-47a5-9559-7917be8270dd\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-djvfd" Oct 30 21:46:07.625002 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.624989 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38375f9a-62ae-4a49-ab8d-f3074b9ac694-workload-certs\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr\" (UID: \"38375f9a-62ae-4a49-ab8d-f3074b9ac694\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr" Oct 30 21:46:07.625213 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.625048 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f74b9971-1030-4e52-8f27-6139e51d8a75-workload-socket\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c\" (UID: \"f74b9971-1030-4e52-8f27-6139e51d8a75\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c" Oct 30 21:46:07.625213 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.625110 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f74b9971-1030-4e52-8f27-6139e51d8a75-credential-socket\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c\" (UID: \"f74b9971-1030-4e52-8f27-6139e51d8a75\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c" Oct 30 21:46:07.625213 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.625130 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/325b45b9-2c8e-47a5-9559-7917be8270dd-istio-envoy\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-djvfd\" (UID: \"325b45b9-2c8e-47a5-9559-7917be8270dd\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-djvfd" Oct 30 21:46:07.625213 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.625158 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/325b45b9-2c8e-47a5-9559-7917be8270dd-istio-podinfo\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-djvfd\" (UID: \"325b45b9-2c8e-47a5-9559-7917be8270dd\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-djvfd" Oct 30 21:46:07.625213 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.625192 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkt6b\" (UniqueName: \"kubernetes.io/projected/38375f9a-62ae-4a49-ab8d-f3074b9ac694-kube-api-access-tkt6b\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr\" (UID: \"38375f9a-62ae-4a49-ab8d-f3074b9ac694\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr" Oct 30 21:46:07.625473 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.625223 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f74b9971-1030-4e52-8f27-6139e51d8a75-workload-certs\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c\" (UID: \"f74b9971-1030-4e52-8f27-6139e51d8a75\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c" Oct 30 21:46:07.726259 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.726168 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38375f9a-62ae-4a49-ab8d-f3074b9ac694-istio-envoy\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr\" (UID: \"38375f9a-62ae-4a49-ab8d-f3074b9ac694\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr" Oct 30 21:46:07.726259 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.726235 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f74b9971-1030-4e52-8f27-6139e51d8a75-istio-podinfo\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c\" (UID: \"f74b9971-1030-4e52-8f27-6139e51d8a75\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c" Oct 30 21:46:07.726527 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.726272 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38375f9a-62ae-4a49-ab8d-f3074b9ac694-istio-podinfo\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr\" (UID: \"38375f9a-62ae-4a49-ab8d-f3074b9ac694\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr" Oct 30 21:46:07.726527 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.726289 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f74b9971-1030-4e52-8f27-6139e51d8a75-istio-envoy\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c\" (UID: \"f74b9971-1030-4e52-8f27-6139e51d8a75\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c" Oct 30 21:46:07.726527 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.726319 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38375f9a-62ae-4a49-ab8d-f3074b9ac694-istio-token\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr\" (UID: \"38375f9a-62ae-4a49-ab8d-f3074b9ac694\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr" Oct 30 21:46:07.726527 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.726346 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/325b45b9-2c8e-47a5-9559-7917be8270dd-credential-socket\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-djvfd\" (UID: \"325b45b9-2c8e-47a5-9559-7917be8270dd\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-djvfd" Oct 30 21:46:07.726527 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.726410 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/325b45b9-2c8e-47a5-9559-7917be8270dd-istiod-ca-cert\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-djvfd\" (UID: \"325b45b9-2c8e-47a5-9559-7917be8270dd\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-djvfd" Oct 30 21:46:07.726527 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.726440 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f74b9971-1030-4e52-8f27-6139e51d8a75-istio-data\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c\" (UID: \"f74b9971-1030-4e52-8f27-6139e51d8a75\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c" Oct 30 21:46:07.726858 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.726751 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp7cz\" (UniqueName: \"kubernetes.io/projected/f74b9971-1030-4e52-8f27-6139e51d8a75-kube-api-access-mp7cz\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c\" (UID: \"f74b9971-1030-4e52-8f27-6139e51d8a75\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c" Oct 30 21:46:07.726858 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.726798 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f74b9971-1030-4e52-8f27-6139e51d8a75-istiod-ca-cert\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c\" (UID: \"f74b9971-1030-4e52-8f27-6139e51d8a75\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c" Oct 30 21:46:07.726858 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.726830 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/325b45b9-2c8e-47a5-9559-7917be8270dd-workload-socket\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-djvfd\" (UID: \"325b45b9-2c8e-47a5-9559-7917be8270dd\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-djvfd" Oct 30 21:46:07.727074 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.726858 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/325b45b9-2c8e-47a5-9559-7917be8270dd-credential-socket\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-djvfd\" (UID: \"325b45b9-2c8e-47a5-9559-7917be8270dd\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-djvfd" Oct 30 21:46:07.727074 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.726865 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f74b9971-1030-4e52-8f27-6139e51d8a75-istio-data\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c\" (UID: \"f74b9971-1030-4e52-8f27-6139e51d8a75\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c" Oct 30 21:46:07.727074 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.726868 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcfgb\" (UniqueName: \"kubernetes.io/projected/325b45b9-2c8e-47a5-9559-7917be8270dd-kube-api-access-gcfgb\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-djvfd\" (UID: \"325b45b9-2c8e-47a5-9559-7917be8270dd\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-djvfd" Oct 30 21:46:07.727074 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.726966 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38375f9a-62ae-4a49-ab8d-f3074b9ac694-workload-certs\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr\" (UID: \"38375f9a-62ae-4a49-ab8d-f3074b9ac694\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr" Oct 30 21:46:07.727074 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.727040 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f74b9971-1030-4e52-8f27-6139e51d8a75-workload-socket\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c\" (UID: \"f74b9971-1030-4e52-8f27-6139e51d8a75\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c" Oct 30 21:46:07.727340 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.727126 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f74b9971-1030-4e52-8f27-6139e51d8a75-credential-socket\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c\" (UID: \"f74b9971-1030-4e52-8f27-6139e51d8a75\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c" Oct 30 21:46:07.727340 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.727154 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/325b45b9-2c8e-47a5-9559-7917be8270dd-istio-envoy\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-djvfd\" (UID: \"325b45b9-2c8e-47a5-9559-7917be8270dd\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-djvfd" Oct 30 21:46:07.727448 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.727408 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/325b45b9-2c8e-47a5-9559-7917be8270dd-workload-socket\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-djvfd\" (UID: \"325b45b9-2c8e-47a5-9559-7917be8270dd\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-djvfd" Oct 30 21:46:07.727448 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.727411 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/325b45b9-2c8e-47a5-9559-7917be8270dd-istiod-ca-cert\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-djvfd\" (UID: \"325b45b9-2c8e-47a5-9559-7917be8270dd\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-djvfd" Oct 30 21:46:07.727561 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.727453 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/325b45b9-2c8e-47a5-9559-7917be8270dd-istio-podinfo\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-djvfd\" (UID: \"325b45b9-2c8e-47a5-9559-7917be8270dd\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-djvfd" Oct 30 21:46:07.727561 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.727497 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkt6b\" (UniqueName: \"kubernetes.io/projected/38375f9a-62ae-4a49-ab8d-f3074b9ac694-kube-api-access-tkt6b\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr\" (UID: \"38375f9a-62ae-4a49-ab8d-f3074b9ac694\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr" Oct 30 21:46:07.727561 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.727532 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f74b9971-1030-4e52-8f27-6139e51d8a75-workload-certs\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c\" (UID: \"f74b9971-1030-4e52-8f27-6139e51d8a75\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c" Oct 30 21:46:07.727717 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.727573 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38375f9a-62ae-4a49-ab8d-f3074b9ac694-istio-data\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr\" (UID: \"38375f9a-62ae-4a49-ab8d-f3074b9ac694\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr" Oct 30 21:46:07.727717 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.727599 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/325b45b9-2c8e-47a5-9559-7917be8270dd-istio-token\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-djvfd\" (UID: \"325b45b9-2c8e-47a5-9559-7917be8270dd\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-djvfd" Oct 30 21:46:07.727717 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.727646 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38375f9a-62ae-4a49-ab8d-f3074b9ac694-istiod-ca-cert\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr\" (UID: \"38375f9a-62ae-4a49-ab8d-f3074b9ac694\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr" Oct 30 21:46:07.727717 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.727664 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f74b9971-1030-4e52-8f27-6139e51d8a75-istiod-ca-cert\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c\" (UID: \"f74b9971-1030-4e52-8f27-6139e51d8a75\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c" Oct 30 21:46:07.727717 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.727683 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/325b45b9-2c8e-47a5-9559-7917be8270dd-workload-certs\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-djvfd\" (UID: \"325b45b9-2c8e-47a5-9559-7917be8270dd\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-djvfd" Oct 30 21:46:07.727717 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.727688 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f74b9971-1030-4e52-8f27-6139e51d8a75-workload-socket\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c\" (UID: \"f74b9971-1030-4e52-8f27-6139e51d8a75\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c" Oct 30 21:46:07.727717 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.727708 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f74b9971-1030-4e52-8f27-6139e51d8a75-istio-token\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c\" (UID: \"f74b9971-1030-4e52-8f27-6139e51d8a75\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c" Oct 30 21:46:07.728079 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.727794 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38375f9a-62ae-4a49-ab8d-f3074b9ac694-workload-socket\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr\" (UID: \"38375f9a-62ae-4a49-ab8d-f3074b9ac694\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr" Oct 30 21:46:07.728079 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.727825 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/325b45b9-2c8e-47a5-9559-7917be8270dd-istio-data\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-djvfd\" (UID: \"325b45b9-2c8e-47a5-9559-7917be8270dd\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-djvfd" Oct 30 21:46:07.728079 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.727858 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38375f9a-62ae-4a49-ab8d-f3074b9ac694-credential-socket\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr\" (UID: \"38375f9a-62ae-4a49-ab8d-f3074b9ac694\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr" Oct 30 21:46:07.728079 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.727926 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f74b9971-1030-4e52-8f27-6139e51d8a75-workload-certs\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c\" (UID: \"f74b9971-1030-4e52-8f27-6139e51d8a75\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c" Oct 30 21:46:07.728079 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.727979 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38375f9a-62ae-4a49-ab8d-f3074b9ac694-workload-certs\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr\" (UID: \"38375f9a-62ae-4a49-ab8d-f3074b9ac694\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr" Oct 30 21:46:07.728387 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.728207 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f74b9971-1030-4e52-8f27-6139e51d8a75-credential-socket\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c\" (UID: \"f74b9971-1030-4e52-8f27-6139e51d8a75\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c" Oct 30 21:46:07.728387 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.728210 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/325b45b9-2c8e-47a5-9559-7917be8270dd-workload-certs\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-djvfd\" (UID: \"325b45b9-2c8e-47a5-9559-7917be8270dd\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-djvfd" Oct 30 21:46:07.728545 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.728515 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38375f9a-62ae-4a49-ab8d-f3074b9ac694-istio-data\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr\" (UID: \"38375f9a-62ae-4a49-ab8d-f3074b9ac694\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr" Oct 30 21:46:07.728990 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.728750 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/325b45b9-2c8e-47a5-9559-7917be8270dd-istio-data\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-djvfd\" (UID: \"325b45b9-2c8e-47a5-9559-7917be8270dd\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-djvfd" Oct 30 21:46:07.728990 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.728848 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38375f9a-62ae-4a49-ab8d-f3074b9ac694-istiod-ca-cert\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr\" (UID: \"38375f9a-62ae-4a49-ab8d-f3074b9ac694\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr" Oct 30 21:46:07.729196 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.729048 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38375f9a-62ae-4a49-ab8d-f3074b9ac694-workload-socket\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr\" (UID: \"38375f9a-62ae-4a49-ab8d-f3074b9ac694\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr" Oct 30 21:46:07.729196 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.729179 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38375f9a-62ae-4a49-ab8d-f3074b9ac694-credential-socket\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr\" (UID: \"38375f9a-62ae-4a49-ab8d-f3074b9ac694\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr" Oct 30 21:46:07.729320 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.729204 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38375f9a-62ae-4a49-ab8d-f3074b9ac694-istio-envoy\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr\" (UID: \"38375f9a-62ae-4a49-ab8d-f3074b9ac694\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr" Oct 30 21:46:07.729556 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.729529 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38375f9a-62ae-4a49-ab8d-f3074b9ac694-istio-podinfo\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr\" (UID: \"38375f9a-62ae-4a49-ab8d-f3074b9ac694\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr" Oct 30 21:46:07.730072 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.730030 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f74b9971-1030-4e52-8f27-6139e51d8a75-istio-envoy\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c\" (UID: \"f74b9971-1030-4e52-8f27-6139e51d8a75\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c" Oct 30 21:46:07.730825 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.730804 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/325b45b9-2c8e-47a5-9559-7917be8270dd-istio-podinfo\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-djvfd\" (UID: \"325b45b9-2c8e-47a5-9559-7917be8270dd\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-djvfd" Oct 30 21:46:07.730990 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.730967 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/325b45b9-2c8e-47a5-9559-7917be8270dd-istio-envoy\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-djvfd\" (UID: \"325b45b9-2c8e-47a5-9559-7917be8270dd\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-djvfd" Oct 30 21:46:07.731287 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.731269 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f74b9971-1030-4e52-8f27-6139e51d8a75-istio-podinfo\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c\" (UID: \"f74b9971-1030-4e52-8f27-6139e51d8a75\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c" Oct 30 21:46:07.788683 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.788643 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38375f9a-62ae-4a49-ab8d-f3074b9ac694-istio-token\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr\" (UID: \"38375f9a-62ae-4a49-ab8d-f3074b9ac694\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr" Oct 30 21:46:07.789170 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.789147 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcfgb\" (UniqueName: \"kubernetes.io/projected/325b45b9-2c8e-47a5-9559-7917be8270dd-kube-api-access-gcfgb\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-djvfd\" (UID: \"325b45b9-2c8e-47a5-9559-7917be8270dd\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-djvfd" Oct 30 21:46:07.789253 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.789167 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp7cz\" (UniqueName: \"kubernetes.io/projected/f74b9971-1030-4e52-8f27-6139e51d8a75-kube-api-access-mp7cz\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c\" (UID: \"f74b9971-1030-4e52-8f27-6139e51d8a75\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c" Oct 30 21:46:07.789492 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.789469 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/325b45b9-2c8e-47a5-9559-7917be8270dd-istio-token\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-djvfd\" (UID: \"325b45b9-2c8e-47a5-9559-7917be8270dd\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-djvfd" Oct 30 21:46:07.790184 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.790167 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f74b9971-1030-4e52-8f27-6139e51d8a75-istio-token\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c\" (UID: \"f74b9971-1030-4e52-8f27-6139e51d8a75\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c" Oct 30 21:46:07.790734 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.790717 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkt6b\" (UniqueName: \"kubernetes.io/projected/38375f9a-62ae-4a49-ab8d-f3074b9ac694-kube-api-access-tkt6b\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr\" (UID: \"38375f9a-62ae-4a49-ab8d-f3074b9ac694\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr" Oct 30 21:46:07.824512 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.824479 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c" Oct 30 21:46:07.833416 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:07.833387 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-djvfd" Oct 30 21:46:08.075445 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:08.075356 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr" Oct 30 21:46:08.265788 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:08.265747 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c"] Oct 30 21:46:08.268821 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:46:08.266783 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf74b9971_1030_4e52_8f27_6139e51d8a75.slice/crio-87b343f07d13bd8787b00e1002851922e1e94b1f3ba72240257a2f77dd77d653 WatchSource:0}: Error finding container 87b343f07d13bd8787b00e1002851922e1e94b1f3ba72240257a2f77dd77d653: Status 404 returned error can't find the container with id 87b343f07d13bd8787b00e1002851922e1e94b1f3ba72240257a2f77dd77d653 Oct 30 21:46:08.268821 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:46:08.267995 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod325b45b9_2c8e_47a5_9559_7917be8270dd.slice/crio-080dcb1860402d0161d70773d9f188bdc5befc4d959eefe444d4bd7ea085ad8c WatchSource:0}: Error finding container 080dcb1860402d0161d70773d9f188bdc5befc4d959eefe444d4bd7ea085ad8c: Status 404 returned error can't find the container with id 080dcb1860402d0161d70773d9f188bdc5befc4d959eefe444d4bd7ea085ad8c Oct 30 21:46:08.271556 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:08.271516 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:46:08.271678 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:08.271642 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:46:08.271740 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:08.271685 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:46:08.272319 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:08.272250 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:46:08.272319 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:08.272304 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:46:08.272458 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:08.272335 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:46:08.272537 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:08.272515 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-djvfd"] Oct 30 21:46:08.466154 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:46:08.466109 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38375f9a_62ae_4a49_ab8d_f3074b9ac694.slice/crio-7ec5004152e6edf30df1c3307c0aede2b3084b91bf53c632fcaf522eb1dca398 WatchSource:0}: Error finding container 7ec5004152e6edf30df1c3307c0aede2b3084b91bf53c632fcaf522eb1dca398: Status 404 returned error can't find the container with id 7ec5004152e6edf30df1c3307c0aede2b3084b91bf53c632fcaf522eb1dca398 Oct 30 21:46:08.467108 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:08.467074 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr"] Oct 30 21:46:08.469253 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:08.469220 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:46:08.469372 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:08.469297 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:46:08.469372 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:08.469347 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:46:08.981702 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:08.981663 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss"] Oct 30 21:46:08.987404 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:08.987378 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss" Oct 30 21:46:08.989412 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:08.989381 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtdx4"] Oct 30 21:46:08.994218 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:08.994195 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtdx4" Oct 30 21:46:09.060392 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.060361 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 21:46:09.060694 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.060674 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 21:46:09.062869 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.062844 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 21:46:09.063183 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.063166 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-9kbqs\"" Oct 30 21:46:09.079056 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.079004 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtdx4"] Oct 30 21:46:09.171386 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.171345 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b589a204-6db6-477e-98ad-362af6f83b79-istiod-ca-cert\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss\" (UID: \"b589a204-6db6-477e-98ad-362af6f83b79\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss" Oct 30 21:46:09.171699 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.171674 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b589a204-6db6-477e-98ad-362af6f83b79-credential-socket\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss\" (UID: \"b589a204-6db6-477e-98ad-362af6f83b79\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss" Oct 30 21:46:09.171878 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.171859 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b589a204-6db6-477e-98ad-362af6f83b79-istio-token\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss\" (UID: \"b589a204-6db6-477e-98ad-362af6f83b79\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss" Oct 30 21:46:09.172177 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.172152 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clmb4\" (UniqueName: \"kubernetes.io/projected/f73607c3-58cc-480e-ba9d-08893ee92244-kube-api-access-clmb4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mtdx4\" (UID: \"f73607c3-58cc-480e-ba9d-08893ee92244\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtdx4" Oct 30 21:46:09.172331 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.172316 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b589a204-6db6-477e-98ad-362af6f83b79-workload-certs\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss\" (UID: \"b589a204-6db6-477e-98ad-362af6f83b79\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss" Oct 30 21:46:09.172439 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.172426 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f73607c3-58cc-480e-ba9d-08893ee92244-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mtdx4\" (UID: \"f73607c3-58cc-480e-ba9d-08893ee92244\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtdx4" Oct 30 21:46:09.172539 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.172527 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b589a204-6db6-477e-98ad-362af6f83b79-istio-podinfo\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss\" (UID: \"b589a204-6db6-477e-98ad-362af6f83b79\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss" Oct 30 21:46:09.172664 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.172650 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f73607c3-58cc-480e-ba9d-08893ee92244-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mtdx4\" (UID: \"f73607c3-58cc-480e-ba9d-08893ee92244\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtdx4" Oct 30 21:46:09.172822 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.172807 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f73607c3-58cc-480e-ba9d-08893ee92244-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mtdx4\" (UID: \"f73607c3-58cc-480e-ba9d-08893ee92244\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtdx4" Oct 30 21:46:09.172954 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.172938 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b589a204-6db6-477e-98ad-362af6f83b79-istio-envoy\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss\" (UID: \"b589a204-6db6-477e-98ad-362af6f83b79\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss" Oct 30 21:46:09.173067 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.173052 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b589a204-6db6-477e-98ad-362af6f83b79-workload-socket\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss\" (UID: \"b589a204-6db6-477e-98ad-362af6f83b79\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss" Oct 30 21:46:09.173194 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.173180 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b589a204-6db6-477e-98ad-362af6f83b79-istio-data\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss\" (UID: \"b589a204-6db6-477e-98ad-362af6f83b79\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss" Oct 30 21:46:09.173794 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.173523 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f73607c3-58cc-480e-ba9d-08893ee92244-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mtdx4\" (UID: \"f73607c3-58cc-480e-ba9d-08893ee92244\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtdx4" Oct 30 21:46:09.174018 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.173998 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f73607c3-58cc-480e-ba9d-08893ee92244-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mtdx4\" (UID: \"f73607c3-58cc-480e-ba9d-08893ee92244\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtdx4" Oct 30 21:46:09.174466 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.174147 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f73607c3-58cc-480e-ba9d-08893ee92244-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mtdx4\" (UID: \"f73607c3-58cc-480e-ba9d-08893ee92244\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtdx4" Oct 30 21:46:09.174651 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.174621 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kttcx\" (UniqueName: \"kubernetes.io/projected/b589a204-6db6-477e-98ad-362af6f83b79-kube-api-access-kttcx\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss\" (UID: \"b589a204-6db6-477e-98ad-362af6f83b79\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss" Oct 30 21:46:09.179985 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.179863 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-djvfd" event={"ID":"325b45b9-2c8e-47a5-9559-7917be8270dd","Type":"ContainerStarted","Data":"6ad4c8c4c9b971250af3415b103221e3926864a02de566b3d8bc4f14f649fcfc"} Oct 30 21:46:09.179985 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.179946 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-djvfd" event={"ID":"325b45b9-2c8e-47a5-9559-7917be8270dd","Type":"ContainerStarted","Data":"080dcb1860402d0161d70773d9f188bdc5befc4d959eefe444d4bd7ea085ad8c"} Oct 30 21:46:09.182859 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.182835 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c" event={"ID":"f74b9971-1030-4e52-8f27-6139e51d8a75","Type":"ContainerStarted","Data":"f73e7169910532cb88013316ae1c816310f7105e2b22a23741ded1e2dc587edb"} Oct 30 21:46:09.182970 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.182879 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c" event={"ID":"f74b9971-1030-4e52-8f27-6139e51d8a75","Type":"ContainerStarted","Data":"87b343f07d13bd8787b00e1002851922e1e94b1f3ba72240257a2f77dd77d653"} Oct 30 21:46:09.184652 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.184631 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr" event={"ID":"38375f9a-62ae-4a49-ab8d-f3074b9ac694","Type":"ContainerStarted","Data":"0411e7eb803150c3d8274fc9fe02c1cef733252134813b5cf3f404ffc54ae3aa"} Oct 30 21:46:09.184755 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.184666 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr" event={"ID":"38375f9a-62ae-4a49-ab8d-f3074b9ac694","Type":"ContainerStarted","Data":"7ec5004152e6edf30df1c3307c0aede2b3084b91bf53c632fcaf522eb1dca398"} Oct 30 21:46:09.243103 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.240503 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss"] Oct 30 21:46:09.276781 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.276743 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f73607c3-58cc-480e-ba9d-08893ee92244-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mtdx4\" (UID: \"f73607c3-58cc-480e-ba9d-08893ee92244\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtdx4" Oct 30 21:46:09.277045 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.277023 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b589a204-6db6-477e-98ad-362af6f83b79-istio-envoy\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss\" (UID: \"b589a204-6db6-477e-98ad-362af6f83b79\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss" Oct 30 21:46:09.277171 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.277155 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b589a204-6db6-477e-98ad-362af6f83b79-workload-socket\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss\" (UID: \"b589a204-6db6-477e-98ad-362af6f83b79\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss" Oct 30 21:46:09.277336 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.277322 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b589a204-6db6-477e-98ad-362af6f83b79-istio-data\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss\" (UID: \"b589a204-6db6-477e-98ad-362af6f83b79\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss" Oct 30 21:46:09.277504 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.277490 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f73607c3-58cc-480e-ba9d-08893ee92244-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mtdx4\" (UID: \"f73607c3-58cc-480e-ba9d-08893ee92244\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtdx4" Oct 30 21:46:09.277664 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.277652 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f73607c3-58cc-480e-ba9d-08893ee92244-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mtdx4\" (UID: \"f73607c3-58cc-480e-ba9d-08893ee92244\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtdx4" Oct 30 21:46:09.277765 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.277752 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f73607c3-58cc-480e-ba9d-08893ee92244-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mtdx4\" (UID: \"f73607c3-58cc-480e-ba9d-08893ee92244\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtdx4" Oct 30 21:46:09.277867 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.277855 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kttcx\" (UniqueName: \"kubernetes.io/projected/b589a204-6db6-477e-98ad-362af6f83b79-kube-api-access-kttcx\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss\" (UID: \"b589a204-6db6-477e-98ad-362af6f83b79\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss" Oct 30 21:46:09.278046 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.278009 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b589a204-6db6-477e-98ad-362af6f83b79-istiod-ca-cert\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss\" (UID: \"b589a204-6db6-477e-98ad-362af6f83b79\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss" Oct 30 21:46:09.278232 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.278214 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b589a204-6db6-477e-98ad-362af6f83b79-credential-socket\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss\" (UID: \"b589a204-6db6-477e-98ad-362af6f83b79\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss" Oct 30 21:46:09.280215 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.280186 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b589a204-6db6-477e-98ad-362af6f83b79-istio-token\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss\" (UID: \"b589a204-6db6-477e-98ad-362af6f83b79\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss" Oct 30 21:46:09.280516 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.280501 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clmb4\" (UniqueName: \"kubernetes.io/projected/f73607c3-58cc-480e-ba9d-08893ee92244-kube-api-access-clmb4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mtdx4\" (UID: \"f73607c3-58cc-480e-ba9d-08893ee92244\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtdx4" Oct 30 21:46:09.280652 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.280637 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b589a204-6db6-477e-98ad-362af6f83b79-workload-certs\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss\" (UID: \"b589a204-6db6-477e-98ad-362af6f83b79\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss" Oct 30 21:46:09.280771 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.280755 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f73607c3-58cc-480e-ba9d-08893ee92244-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mtdx4\" (UID: \"f73607c3-58cc-480e-ba9d-08893ee92244\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtdx4" Oct 30 21:46:09.280881 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.280869 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b589a204-6db6-477e-98ad-362af6f83b79-istio-podinfo\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss\" (UID: \"b589a204-6db6-477e-98ad-362af6f83b79\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss" Oct 30 21:46:09.281060 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.281027 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f73607c3-58cc-480e-ba9d-08893ee92244-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mtdx4\" (UID: \"f73607c3-58cc-480e-ba9d-08893ee92244\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtdx4" Oct 30 21:46:09.285361 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.285333 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b589a204-6db6-477e-98ad-362af6f83b79-istiod-ca-cert\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss\" (UID: \"b589a204-6db6-477e-98ad-362af6f83b79\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss" Oct 30 21:46:09.288980 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.288953 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b589a204-6db6-477e-98ad-362af6f83b79-credential-socket\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss\" (UID: \"b589a204-6db6-477e-98ad-362af6f83b79\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss" Oct 30 21:46:09.290132 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.290109 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b589a204-6db6-477e-98ad-362af6f83b79-workload-certs\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss\" (UID: \"b589a204-6db6-477e-98ad-362af6f83b79\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss" Oct 30 21:46:09.291218 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.291187 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b589a204-6db6-477e-98ad-362af6f83b79-workload-socket\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss\" (UID: \"b589a204-6db6-477e-98ad-362af6f83b79\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss" Oct 30 21:46:09.293028 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.291706 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b589a204-6db6-477e-98ad-362af6f83b79-istio-data\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss\" (UID: \"b589a204-6db6-477e-98ad-362af6f83b79\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss" Oct 30 21:46:09.295281 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.295254 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f73607c3-58cc-480e-ba9d-08893ee92244-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mtdx4\" (UID: \"f73607c3-58cc-480e-ba9d-08893ee92244\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtdx4" Oct 30 21:46:09.296701 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.296377 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b589a204-6db6-477e-98ad-362af6f83b79-istio-podinfo\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss\" (UID: \"b589a204-6db6-477e-98ad-362af6f83b79\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss" Oct 30 21:46:09.298350 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.298321 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f73607c3-58cc-480e-ba9d-08893ee92244-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mtdx4\" (UID: \"f73607c3-58cc-480e-ba9d-08893ee92244\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtdx4" Oct 30 21:46:09.298450 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.298327 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b589a204-6db6-477e-98ad-362af6f83b79-istio-envoy\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss\" (UID: \"b589a204-6db6-477e-98ad-362af6f83b79\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss" Oct 30 21:46:09.299015 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.298989 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f73607c3-58cc-480e-ba9d-08893ee92244-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mtdx4\" (UID: \"f73607c3-58cc-480e-ba9d-08893ee92244\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtdx4" Oct 30 21:46:09.299096 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.299034 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f73607c3-58cc-480e-ba9d-08893ee92244-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mtdx4\" (UID: \"f73607c3-58cc-480e-ba9d-08893ee92244\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtdx4" Oct 30 21:46:09.304984 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.299466 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f73607c3-58cc-480e-ba9d-08893ee92244-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mtdx4\" (UID: \"f73607c3-58cc-480e-ba9d-08893ee92244\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtdx4" Oct 30 21:46:09.365922 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.359641 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f73607c3-58cc-480e-ba9d-08893ee92244-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mtdx4\" (UID: \"f73607c3-58cc-480e-ba9d-08893ee92244\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtdx4" Oct 30 21:46:09.365922 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.363078 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kttcx\" (UniqueName: \"kubernetes.io/projected/b589a204-6db6-477e-98ad-362af6f83b79-kube-api-access-kttcx\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss\" (UID: \"b589a204-6db6-477e-98ad-362af6f83b79\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss" Oct 30 21:46:09.372880 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.372820 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clmb4\" (UniqueName: \"kubernetes.io/projected/f73607c3-58cc-480e-ba9d-08893ee92244-kube-api-access-clmb4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mtdx4\" (UID: \"f73607c3-58cc-480e-ba9d-08893ee92244\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtdx4" Oct 30 21:46:09.373331 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.373304 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b589a204-6db6-477e-98ad-362af6f83b79-istio-token\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss\" (UID: \"b589a204-6db6-477e-98ad-362af6f83b79\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss" Oct 30 21:46:09.479626 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.479568 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-djvfd" podStartSLOduration=2.479548345 podStartE2EDuration="2.479548345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:09.453434681 +0000 UTC m=+1353.195955616" watchObservedRunningTime="2026-10-30 21:46:09.479548345 +0000 UTC m=+1353.222069286" Oct 30 21:46:09.638092 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.638060 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtdx4" Oct 30 21:46:09.639243 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.639216 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss" Oct 30 21:46:09.773004 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.763017 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c" podStartSLOduration=2.762992875 podStartE2EDuration="2.762992875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:09.638867287 +0000 UTC m=+1353.381388257" watchObservedRunningTime="2026-10-30 21:46:09.762992875 +0000 UTC m=+1353.505513815" Oct 30 21:46:09.855800 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.835986 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-djvfd" Oct 30 21:46:09.855800 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.837040 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c" Oct 30 21:46:09.855800 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.838592 5748 patch_prober.go:29] interesting pod/e2e-worker-w5-c7d477-openshift-default-776595695-djvfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 30 21:46:09.855800 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.838655 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-djvfd" podUID="325b45b9-2c8e-47a5-9559-7917be8270dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 30 21:46:09.856414 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.856283 5748 patch_prober.go:29] interesting pod/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 21:46:09.856414 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:09.856346 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c" podUID="f74b9971-1030-4e52-8f27-6139e51d8a75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 21:46:10.085944 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:10.083157 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr" Oct 30 21:46:10.085944 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:10.083294 5748 patch_prober.go:29] interesting pod/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 21:46:10.085944 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:10.083355 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr" podUID="38375f9a-62ae-4a49-ab8d-f3074b9ac694" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 21:46:10.790639 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:10.790580 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr" podStartSLOduration=3.790558683 podStartE2EDuration="3.790558683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:09.765500997 +0000 UTC m=+1353.508021932" watchObservedRunningTime="2026-10-30 21:46:10.790558683 +0000 UTC m=+1354.533079620" Oct 30 21:46:10.794452 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:10.794425 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss"] Oct 30 21:46:10.798354 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:10.798321 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:46:10.798473 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:10.798402 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:46:10.798473 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:10.798451 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:46:10.886141 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:10.885014 5748 patch_prober.go:29] interesting pod/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 21:46:10.886141 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:10.885066 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c" podUID="f74b9971-1030-4e52-8f27-6139e51d8a75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 21:46:10.886141 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:10.885367 5748 patch_prober.go:29] interesting pod/e2e-worker-w5-c7d477-openshift-default-776595695-djvfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 30 21:46:10.886141 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:10.885425 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-djvfd" podUID="325b45b9-2c8e-47a5-9559-7917be8270dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 30 21:46:11.087234 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:46:11.087071 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf73607c3_58cc_480e_ba9d_08893ee92244.slice/crio-bf03a221b7ae9e84d17ff1d90832ab10af6b56d5068a2b46b325452cf9878e89 WatchSource:0}: Error finding container bf03a221b7ae9e84d17ff1d90832ab10af6b56d5068a2b46b325452cf9878e89: Status 404 returned error can't find the container with id bf03a221b7ae9e84d17ff1d90832ab10af6b56d5068a2b46b325452cf9878e89 Oct 30 21:46:11.087234 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:11.087096 5748 patch_prober.go:29] interesting pod/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 21:46:11.087234 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:11.087171 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr" podUID="38375f9a-62ae-4a49-ab8d-f3074b9ac694" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 21:46:11.089475 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:11.089450 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtdx4"] Oct 30 21:46:11.089660 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:11.089640 5748 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:46:11.292923 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:11.292623 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss" event={"ID":"b589a204-6db6-477e-98ad-362af6f83b79","Type":"ContainerStarted","Data":"377bab6ee135cc461038266457bea5702b23a0426fae0b1a5467f0a2d38a0781"} Oct 30 21:46:11.292923 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:11.292709 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss" event={"ID":"b589a204-6db6-477e-98ad-362af6f83b79","Type":"ContainerStarted","Data":"c7e785c86513eb692053e58fbc8658aa5d832be0b76802d1b6c90e54cf323026"} Oct 30 21:46:11.295457 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:11.295427 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtdx4" event={"ID":"f73607c3-58cc-480e-ba9d-08893ee92244","Type":"ContainerStarted","Data":"bf03a221b7ae9e84d17ff1d90832ab10af6b56d5068a2b46b325452cf9878e89"} Oct 30 21:46:11.454251 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:11.454186 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss" podStartSLOduration=3.454150138 podStartE2EDuration="3.454150138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:11.433390952 +0000 UTC m=+1355.175911888" watchObservedRunningTime="2026-10-30 21:46:11.454150138 +0000 UTC m=+1355.196671074" Oct 30 21:46:11.667043 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:11.666957 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss" Oct 30 21:46:11.844284 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:11.844247 5748 patch_prober.go:29] interesting pod/e2e-worker-w5-c7d477-openshift-default-776595695-djvfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 30 21:46:11.845608 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:11.844826 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-djvfd" podUID="325b45b9-2c8e-47a5-9559-7917be8270dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 30 21:46:11.845608 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:11.845524 5748 patch_prober.go:29] interesting pod/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 21:46:11.845608 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:11.845563 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c" podUID="f74b9971-1030-4e52-8f27-6139e51d8a75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 21:46:12.116008 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:12.115628 5748 patch_prober.go:29] interesting pod/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 21:46:12.116008 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:12.115692 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr" podUID="38375f9a-62ae-4a49-ab8d-f3074b9ac694" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 21:46:12.689142 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:12.689090 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ea374/simulator-w1-6ea374-kserve-775cfdf98b-p5q25" podUID="b0d53207-65f3-4d43-baba-66bc4c366885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:12.774924 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:12.751860 5748 patch_prober.go:29] interesting pod/e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 30 21:46:12.774924 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:12.751942 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss" podUID="b589a204-6db6-477e-98ad-362af6f83b79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 30 21:46:12.832986 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:12.832932 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-69e4b2/premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b" podUID="3ed43145-33bd-4c11-bb29-dc852511f6d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:12.839222 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:12.839137 5748 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-c7d477/distinct-w5-c7d477-kserve-554d7f974d-6zw6s" podUID="09d39cc0-b68d-48fd-918c-7d2518f384aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:12.866927 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:12.841983 5748 patch_prober.go:29] interesting pod/e2e-worker-w5-c7d477-openshift-default-776595695-djvfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 30 21:46:12.866927 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:12.842032 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-djvfd" podUID="325b45b9-2c8e-47a5-9559-7917be8270dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 30 21:46:12.958730 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:12.958632 5748 patch_prober.go:29] interesting pod/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 21:46:12.959023 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:12.958973 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c" podUID="f74b9971-1030-4e52-8f27-6139e51d8a75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 21:46:13.403980 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:13.301179 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-69e4b2/simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj" podUID="698fc0f2-64ce-4e7e-b482-8ebd9f33f84d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:13.408266 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:13.408224 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-691598/simulator-w0-691598-kserve-76fb5d5bc7-l5gqk" podUID="4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 30 21:46:13.466033 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:13.447541 5748 patch_prober.go:29] interesting pod/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 21:46:13.709110 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:13.669007 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h" podUID="0a0b0425-2959-4690-9847-38852e167230" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Oct 30 21:46:13.709474 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:13.642009 5748 patch_prober.go:29] interesting pod/e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 30 21:46:13.709672 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:13.709643 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss" podUID="b589a204-6db6-477e-98ad-362af6f83b79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 30 21:46:13.777925 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:13.670066 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c7d477/distinct-w5-c7d477-kserve-554d7f974d-6zw6s" podUID="09d39cc0-b68d-48fd-918c-7d2518f384aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:13.777925 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:13.723144 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ea374/distinct-2-w1-6ea374-kserve-68fb49889-hpg2k" podUID="ab533734-e0d8-4acb-854d-12851a87bc58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:13.777925 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:13.723310 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr" podUID="38375f9a-62ae-4a49-ab8d-f3074b9ac694" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 21:46:13.777925 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:13.725722 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ea374/simulator-w1-6ea374-kserve-775cfdf98b-p5q25" podUID="b0d53207-65f3-4d43-baba-66bc4c366885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 21:46:13.779992 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:13.779945 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91874e/distinct-2-w2-91874e-kserve-69fcf64f44-qm96r" podUID="0cb16f33-b208-4c11-bb12-322829bd51e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:13.837922 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:13.726888 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6ea374/simulator-w1-6ea374-kserve-775cfdf98b-p5q25" Oct 30 21:46:13.912921 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:13.866807 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d68156/simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk" podUID="db285e9c-ca87-43fa-8f06-69e271e5c2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 30 21:46:13.917918 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:13.875007 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91874e/distinct-w2-91874e-kserve-58d6bdfc75-bdp5z" podUID="cb39b415-f406-47bd-b591-be7e09240f5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:13.961711 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:13.961628 5748 patch_prober.go:29] interesting pod/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 21:46:13.961963 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:13.961933 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c" podUID="f74b9971-1030-4e52-8f27-6139e51d8a75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 21:46:13.962153 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:13.962137 5748 patch_prober.go:29] interesting pod/e2e-worker-w5-c7d477-openshift-default-776595695-djvfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 30 21:46:13.962272 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:13.962254 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-djvfd" podUID="325b45b9-2c8e-47a5-9559-7917be8270dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 30 21:46:14.245979 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:14.223035 5748 patch_prober.go:29] interesting pod/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 21:46:14.245979 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:14.223101 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr" podUID="38375f9a-62ae-4a49-ab8d-f3074b9ac694" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 21:46:14.347922 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:14.303048 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-69e4b2/simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj" podUID="698fc0f2-64ce-4e7e-b482-8ebd9f33f84d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:14.357452 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:14.357417 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-69e4b2/simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj" Oct 30 21:46:14.416012 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:14.415968 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-691598/simulator-w0-691598-kserve-76fb5d5bc7-l5gqk" podUID="4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:14.581983 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:14.575651 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-691598/simulator-w0-691598-kserve-76fb5d5bc7-l5gqk" Oct 30 21:46:14.639929 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:14.610027 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d68156/distinct-w3-d68156-kserve-565894fdb4-85t7z" podUID="13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:14.673929 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:14.650128 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6ea374/simulator-w1-6ea374-kserve-775cfdf98b-p5q25" Oct 30 21:46:14.673929 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:14.673576 5748 patch_prober.go:29] interesting pod/e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 30 21:46:14.673929 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:14.673631 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss" podUID="b589a204-6db6-477e-98ad-362af6f83b79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 30 21:46:14.731926 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:14.714232 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h" podUID="0a0b0425-2959-4690-9847-38852e167230" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Oct 30 21:46:14.731926 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:14.714673 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h" Oct 30 21:46:14.736424 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:14.736384 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c7d477/distinct-w5-c7d477-kserve-554d7f974d-6zw6s" podUID="09d39cc0-b68d-48fd-918c-7d2518f384aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Oct 30 21:46:14.737123 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:14.737098 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c7d477/distinct-w5-c7d477-kserve-554d7f974d-6zw6s" Oct 30 21:46:14.745926 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:14.738314 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ea374/distinct-2-w1-6ea374-kserve-68fb49889-hpg2k" podUID="ab533734-e0d8-4acb-854d-12851a87bc58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 30 21:46:14.767923 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:14.764445 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6ea374/distinct-2-w1-6ea374-kserve-68fb49889-hpg2k" Oct 30 21:46:14.983926 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:14.783479 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91874e/distinct-2-w2-91874e-kserve-69fcf64f44-qm96r" podUID="0cb16f33-b208-4c11-bb12-322829bd51e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Oct 30 21:46:14.983926 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:14.935874 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91874e/distinct-w2-91874e-kserve-58d6bdfc75-bdp5z" podUID="cb39b415-f406-47bd-b591-be7e09240f5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 30 21:46:14.983926 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:14.977693 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-91874e/distinct-2-w2-91874e-kserve-69fcf64f44-qm96r" Oct 30 21:46:14.983926 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:14.977748 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-91874e/distinct-w2-91874e-kserve-58d6bdfc75-bdp5z" Oct 30 21:46:14.983926 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:14.947230 5748 patch_prober.go:29] interesting pod/e2e-worker-w5-c7d477-openshift-default-776595695-djvfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 30 21:46:14.983926 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:14.978623 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-djvfd" podUID="325b45b9-2c8e-47a5-9559-7917be8270dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 30 21:46:14.983926 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:14.947290 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d68156/simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk" podUID="db285e9c-ca87-43fa-8f06-69e271e5c2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 30 21:46:14.983926 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:14.979584 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d68156/simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk" Oct 30 21:46:14.986861 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:14.986833 5748 patch_prober.go:29] interesting pod/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 21:46:15.042023 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:15.041971 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c" podUID="f74b9971-1030-4e52-8f27-6139e51d8a75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 21:46:15.050042 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:15.049992 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-69e4b2/premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b" podUID="3ed43145-33bd-4c11-bb29-dc852511f6d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:15.121922 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:15.109673 5748 patch_prober.go:29] interesting pod/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 21:46:15.121922 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:15.109737 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr" podUID="38375f9a-62ae-4a49-ab8d-f3074b9ac694" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 21:46:15.412929 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:15.410174 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-69e4b2/simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj" podUID="698fc0f2-64ce-4e7e-b482-8ebd9f33f84d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 30 21:46:15.445990 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:15.428618 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-691598/simulator-w0-691598-kserve-76fb5d5bc7-l5gqk" podUID="4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:15.445990 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:15.431925 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c7d477/distinct-w5-c7d477-kserve-554d7f974d-6zw6s" Oct 30 21:46:15.561604 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:15.561563 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1de75a/simulator-w4-1de75a-kserve-6887dd4b46-kc5c2" podUID="926fa847-0c3c-43e0-b378-04c82433cd98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:15.565813 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:15.565786 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-691598/simulator-w0-691598-kserve-76fb5d5bc7-l5gqk" Oct 30 21:46:15.585917 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:15.570176 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-91874e/distinct-w2-91874e-kserve-58d6bdfc75-bdp5z" Oct 30 21:46:15.585917 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:15.572399 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-69e4b2/simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj" Oct 30 21:46:15.654922 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:15.647976 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d68156/distinct-w3-d68156-kserve-565894fdb4-85t7z" podUID="13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:15.672464 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:15.655519 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d68156/distinct-w3-d68156-kserve-565894fdb4-85t7z" Oct 30 21:46:15.672464 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:15.657021 5748 patch_prober.go:29] interesting pod/e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 30 21:46:15.672464 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:15.657059 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss" podUID="b589a204-6db6-477e-98ad-362af6f83b79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 30 21:46:15.672464 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:15.657943 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6ea374/distinct-2-w1-6ea374-kserve-68fb49889-hpg2k" Oct 30 21:46:15.686798 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:15.686770 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d68156/distinct-w3-d68156-kserve-565894fdb4-85t7z" Oct 30 21:46:15.687085 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:15.687066 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h" Oct 30 21:46:15.724435 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:15.724409 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-91874e/distinct-2-w2-91874e-kserve-69fcf64f44-qm96r" Oct 30 21:46:15.726683 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:15.726662 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d68156/simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk" Oct 30 21:46:15.834385 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:15.834347 5748 patch_prober.go:29] interesting pod/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 21:46:15.836053 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:15.836017 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c" podUID="f74b9971-1030-4e52-8f27-6139e51d8a75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 21:46:15.836291 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:15.834913 5748 patch_prober.go:29] interesting pod/e2e-worker-w5-c7d477-openshift-default-776595695-djvfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 30 21:46:15.836435 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:15.836415 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-djvfd" podUID="325b45b9-2c8e-47a5-9559-7917be8270dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 30 21:46:16.076771 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:16.076645 5748 patch_prober.go:29] interesting pod/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 21:46:16.076771 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:16.076709 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr" podUID="38375f9a-62ae-4a49-ab8d-f3074b9ac694" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 21:46:16.644924 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:16.644648 5748 patch_prober.go:29] interesting pod/e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 30 21:46:16.644924 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:16.644721 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss" podUID="b589a204-6db6-477e-98ad-362af6f83b79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 30 21:46:17.476936 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:17.452162 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-djvfd" Oct 30 21:46:17.476936 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:17.475032 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-djvfd" Oct 30 21:46:17.508641 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:17.506393 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-djvfd" Oct 30 21:46:17.599776 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:17.599740 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr" Oct 30 21:46:17.609598 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:17.601890 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr" Oct 30 21:46:17.642274 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:17.611152 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr" Oct 30 21:46:17.661953 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:17.646418 5748 patch_prober.go:29] interesting pod/e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 30 21:46:17.661953 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:17.646478 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss" podUID="b589a204-6db6-477e-98ad-362af6f83b79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 30 21:46:17.661953 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:17.646807 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c" Oct 30 21:46:17.661953 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:17.654459 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c" Oct 30 21:46:17.661953 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:17.661557 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c" Oct 30 21:46:18.644137 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:18.644095 5748 patch_prober.go:29] interesting pod/e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 30 21:46:18.644881 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:18.644195 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss" podUID="b589a204-6db6-477e-98ad-362af6f83b79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 30 21:46:19.847991 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:19.847793 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss" Oct 30 21:46:19.848961 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:19.848886 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss" Oct 30 21:46:19.849782 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:19.849760 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss" Oct 30 21:46:20.392428 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:20.392395 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:46:20.392519 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:20.392462 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:46:20.785282 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:20.785198 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtdx4" event={"ID":"f73607c3-58cc-480e-ba9d-08893ee92244","Type":"ContainerStarted","Data":"1ad4f8c4ffeeed83d926fe082f25359a4ce8f0bec2fd821ca6c052b72394cd03"} Oct 30 21:46:20.785591 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:20.785571 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtdx4" Oct 30 21:46:20.912911 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:20.912828 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtdx4" podStartSLOduration=3.611563226 podStartE2EDuration="12.912811189s" totalImagesPullingTime="9.301247963s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:20.908047837 +0000 UTC m=+1364.650568775" watchObservedRunningTime="2026-10-30 21:46:20.912811189 +0000 UTC m=+1364.655332124" Oct 30 21:46:21.798510 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:21.798448 5748 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-mtdx4 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:46:21.798767 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:21.798519 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtdx4" podUID="f73607c3-58cc-480e-ba9d-08893ee92244" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:46:24.797496 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:24.797463 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtdx4" Oct 30 21:46:28.487921 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:28.477215 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1de75a/simulator-w4-1de75a-kserve-6887dd4b46-kc5c2" podUID="926fa847-0c3c-43e0-b378-04c82433cd98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Oct 30 21:46:28.585724 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:28.585669 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-69e4b2/simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj" podUID="698fc0f2-64ce-4e7e-b482-8ebd9f33f84d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:28.586351 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:28.586320 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91874e/distinct-w2-91874e-kserve-58d6bdfc75-bdp5z" podUID="cb39b415-f406-47bd-b591-be7e09240f5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 30 21:46:28.656028 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:28.655976 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ea374/simulator-w1-6ea374-kserve-775cfdf98b-p5q25" podUID="b0d53207-65f3-4d43-baba-66bc4c366885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 21:46:28.698059 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:28.698003 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ea374/distinct-2-w1-6ea374-kserve-68fb49889-hpg2k" podUID="ab533734-e0d8-4acb-854d-12851a87bc58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:28.699212 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:28.699178 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d68156/distinct-w3-d68156-kserve-565894fdb4-85t7z" podUID="13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:28.745639 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:28.745532 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91874e/distinct-2-w2-91874e-kserve-69fcf64f44-qm96r" podUID="0cb16f33-b208-4c11-bb12-322829bd51e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:28.778924 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:28.778864 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h" podUID="0a0b0425-2959-4690-9847-38852e167230" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Oct 30 21:46:28.780027 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:28.779996 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-69e4b2/premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b" podUID="3ed43145-33bd-4c11-bb29-dc852511f6d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 21:46:28.797057 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:28.797000 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d68156/simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk" podUID="db285e9c-ca87-43fa-8f06-69e271e5c2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:28.880071 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:28.880022 5748 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h" podUID="0a0b0425-2959-4690-9847-38852e167230" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:29.436922 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:29.436001 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c7d477/distinct-w5-c7d477-kserve-554d7f974d-6zw6s" podUID="09d39cc0-b68d-48fd-918c-7d2518f384aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:29.440922 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:29.437282 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-691598/simulator-w0-691598-kserve-76fb5d5bc7-l5gqk" podUID="4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:29.481054 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:29.480974 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1de75a/simulator-w4-1de75a-kserve-6887dd4b46-kc5c2" podUID="926fa847-0c3c-43e0-b378-04c82433cd98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Oct 30 21:46:29.481721 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:29.481696 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1de75a/simulator-w4-1de75a-kserve-6887dd4b46-kc5c2" Oct 30 21:46:29.539106 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:29.539058 5748 patch_prober.go:29] interesting pod/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:46:29.539689 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:29.539642 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd" podUID="890f0ec3-96a5-4ea5-8137-e7166101e49b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:29.710035 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:29.707189 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d68156/distinct-w3-d68156-kserve-565894fdb4-85t7z" podUID="13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:29.711592 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:29.711555 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ea374/simulator-w1-6ea374-kserve-775cfdf98b-p5q25" podUID="b0d53207-65f3-4d43-baba-66bc4c366885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 21:46:29.740921 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:29.712064 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ea374/distinct-2-w1-6ea374-kserve-68fb49889-hpg2k" podUID="ab533734-e0d8-4acb-854d-12851a87bc58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 30 21:46:29.747488 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:29.725437 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d68156/distinct-w3-d68156-kserve-565894fdb4-85t7z" Oct 30 21:46:29.747779 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:29.727316 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-69e4b2/simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj" podUID="698fc0f2-64ce-4e7e-b482-8ebd9f33f84d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 30 21:46:29.747984 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:29.727347 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91874e/distinct-w2-91874e-kserve-58d6bdfc75-bdp5z" podUID="cb39b415-f406-47bd-b591-be7e09240f5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 30 21:46:29.748185 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:29.746058 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91874e/distinct-2-w2-91874e-kserve-69fcf64f44-qm96r" podUID="0cb16f33-b208-4c11-bb12-322829bd51e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Oct 30 21:46:29.748394 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:29.748380 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6ea374/simulator-w1-6ea374-kserve-775cfdf98b-p5q25" Oct 30 21:46:29.757704 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:29.750388 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6ea374/distinct-2-w1-6ea374-kserve-68fb49889-hpg2k" Oct 30 21:46:29.757704 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:29.750450 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-69e4b2/simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj" Oct 30 21:46:29.757704 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:29.750488 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-91874e/distinct-w2-91874e-kserve-58d6bdfc75-bdp5z" Oct 30 21:46:29.757704 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:29.750531 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-91874e/distinct-2-w2-91874e-kserve-69fcf64f44-qm96r" Oct 30 21:46:29.795204 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:29.791950 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-69e4b2/premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b" podUID="3ed43145-33bd-4c11-bb29-dc852511f6d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:29.795204 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:29.792059 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h" podUID="0a0b0425-2959-4690-9847-38852e167230" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Oct 30 21:46:29.813273 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:29.806837 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-69e4b2/premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b" Oct 30 21:46:29.813273 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:29.806924 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h" Oct 30 21:46:29.813273 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:29.808479 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d68156/simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk" podUID="db285e9c-ca87-43fa-8f06-69e271e5c2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:29.813273 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:29.808813 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d68156/simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk" Oct 30 21:46:30.144418 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:30.144371 5748 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-d68156/simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk" podUID="db285e9c-ca87-43fa-8f06-69e271e5c2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:30.460667 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:30.460123 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c7d477/distinct-w5-c7d477-kserve-554d7f974d-6zw6s" podUID="09d39cc0-b68d-48fd-918c-7d2518f384aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Oct 30 21:46:30.468661 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:30.460079 5748 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6ea374/distinct-2-w1-6ea374-kserve-68fb49889-hpg2k" podUID="ab533734-e0d8-4acb-854d-12851a87bc58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": context deadline exceeded" Oct 30 21:46:30.481504 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:30.481435 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-691598/simulator-w0-691598-kserve-76fb5d5bc7-l5gqk" podUID="4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 30 21:46:30.494945 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:30.482960 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1de75a/simulator-w4-1de75a-kserve-6887dd4b46-kc5c2" podUID="926fa847-0c3c-43e0-b378-04c82433cd98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:30.593920 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:30.583669 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c7d477/distinct-w5-c7d477-kserve-554d7f974d-6zw6s" Oct 30 21:46:30.595244 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:30.595203 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-691598/simulator-w0-691598-kserve-76fb5d5bc7-l5gqk" Oct 30 21:46:30.710041 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:30.709993 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d68156/distinct-w3-d68156-kserve-565894fdb4-85t7z" podUID="13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:30.756218 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:30.756129 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91874e/distinct-2-w2-91874e-kserve-69fcf64f44-qm96r" podUID="0cb16f33-b208-4c11-bb12-322829bd51e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:30.761783 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:30.760715 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91874e/distinct-w2-91874e-kserve-58d6bdfc75-bdp5z" podUID="cb39b415-f406-47bd-b591-be7e09240f5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 30 21:46:30.790123 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:30.775056 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ea374/distinct-2-w1-6ea374-kserve-68fb49889-hpg2k" podUID="ab533734-e0d8-4acb-854d-12851a87bc58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:30.790123 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:30.775369 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-69e4b2/simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj" podUID="698fc0f2-64ce-4e7e-b482-8ebd9f33f84d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 30 21:46:30.790123 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:30.777129 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ea374/simulator-w1-6ea374-kserve-775cfdf98b-p5q25" podUID="b0d53207-65f3-4d43-baba-66bc4c366885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 21:46:30.814307 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:30.810766 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d68156/simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk" podUID="db285e9c-ca87-43fa-8f06-69e271e5c2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 30 21:46:30.814307 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:30.812128 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-69e4b2/premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b" podUID="3ed43145-33bd-4c11-bb29-dc852511f6d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 21:46:30.814307 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:30.813126 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h" podUID="0a0b0425-2959-4690-9847-38852e167230" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Oct 30 21:46:30.840633 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:30.840603 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1de75a/simulator-w4-1de75a-kserve-6887dd4b46-kc5c2" Oct 30 21:46:30.861032 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:46:30.860977 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:46:30.985434 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:30.985386 5748 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-91874e/distinct-2-w2-91874e-kserve-69fcf64f44-qm96r" podUID="0cb16f33-b208-4c11-bb12-322829bd51e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:31.112045 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:31.104442 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-91874e/distinct-2-w2-91874e-kserve-69fcf64f44-qm96r" Oct 30 21:46:31.156169 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:31.156126 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h" Oct 30 21:46:31.171422 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:31.171368 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-91874e/distinct-w2-91874e-kserve-58d6bdfc75-bdp5z" Oct 30 21:46:31.222810 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:31.222766 5748 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-691598/simulator-w0-691598-kserve-76fb5d5bc7-l5gqk" podUID="4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": context deadline exceeded" Oct 30 21:46:31.324221 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:31.324191 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6ea374/simulator-w1-6ea374-kserve-775cfdf98b-p5q25" Oct 30 21:46:31.324499 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:31.324479 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d68156/distinct-w3-d68156-kserve-565894fdb4-85t7z" Oct 30 21:46:31.473920 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:31.460645 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c7d477/distinct-w5-c7d477-kserve-554d7f974d-6zw6s" podUID="09d39cc0-b68d-48fd-918c-7d2518f384aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Oct 30 21:46:31.681184 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:31.681129 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-69e4b2/simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj" Oct 30 21:46:31.732793 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:31.732609 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-691598/simulator-w0-691598-kserve-76fb5d5bc7-l5gqk" podUID="4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 30 21:46:31.742328 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:31.742296 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c7d477/distinct-w5-c7d477-kserve-554d7f974d-6zw6s" Oct 30 21:46:31.743620 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:31.743595 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d68156/simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk" Oct 30 21:46:31.744657 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:31.744614 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6ea374/distinct-2-w1-6ea374-kserve-68fb49889-hpg2k" Oct 30 21:46:31.762631 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:31.749619 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-69e4b2/premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b" Oct 30 21:46:31.781124 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:31.781096 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-691598/simulator-w0-691598-kserve-76fb5d5bc7-l5gqk" Oct 30 21:46:36.846376 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:36.846345 5748 scope.go:122] "RemoveContainer" containerID="ad7215a71168bd6f91c9f16b06d9d0ed85af38dd506472da47ff2d8045180a1e" Oct 30 21:46:36.857198 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:36.857177 5748 scope.go:122] "RemoveContainer" containerID="fcb05ff7dc55df4ac2a9356ec89369b5d3876b2dc0e75d958ba5f1f88aac4065" Oct 30 21:46:36.866827 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:36.866809 5748 scope.go:122] "RemoveContainer" containerID="ef6132ff756c537ef6be6af76156e32979c1b73dfe271127d485ddcfc7e64662" Oct 30 21:46:36.875866 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:36.875847 5748 scope.go:122] "RemoveContainer" containerID="605068d78273e034e494ce8cef9bbb2519eec529dd7dd3de01802f777731c782" Oct 30 21:46:36.885175 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:36.885153 5748 scope.go:122] "RemoveContainer" containerID="4d9ab051c9273ee2a695b9da12a63e55226755c79e4e88b4949dd5e6af5f7b3a" Oct 30 21:46:36.894114 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:36.894093 5748 scope.go:122] "RemoveContainer" containerID="6b9d86defbfb58f739f12842e5b15c7e0e3e20e6b3be40f58de94a53c29a5ec8" Oct 30 21:46:36.903317 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:36.903294 5748 scope.go:122] "RemoveContainer" containerID="5e9071b82ec895db26e20b5edf4ec7213468e4147a31768d0b9158e3fd784053" Oct 30 21:46:36.912452 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:46:36.912432 5748 scope.go:122] "RemoveContainer" containerID="278c01d11609244128c36df28a970d20fe331901701f8b45a151a73141d8cdbf" Oct 30 21:47:00.157328 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:47:00.157289 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:47:02.954049 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:02.954003 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-98fnd"] Oct 30 21:47:02.962120 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:02.962094 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-98fnd" Oct 30 21:47:03.068724 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:03.068690 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-98fnd\" (UID: \"d1c1a2b1-f312-47b1-b437-2d8f1760eea0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-98fnd" Oct 30 21:47:03.068919 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:03.068740 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-98fnd\" (UID: \"d1c1a2b1-f312-47b1-b437-2d8f1760eea0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-98fnd" Oct 30 21:47:03.068981 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:03.068939 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-98fnd\" (UID: \"d1c1a2b1-f312-47b1-b437-2d8f1760eea0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-98fnd" Oct 30 21:47:03.069114 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:03.069094 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-98fnd\" (UID: \"d1c1a2b1-f312-47b1-b437-2d8f1760eea0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-98fnd" Oct 30 21:47:03.069170 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:03.069151 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lv7l\" (UniqueName: \"kubernetes.io/projected/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-kube-api-access-6lv7l\") pod \"maas-default-gateway-openshift-default-75d7c785f-98fnd\" (UID: \"d1c1a2b1-f312-47b1-b437-2d8f1760eea0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-98fnd" Oct 30 21:47:03.069293 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:03.069273 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-98fnd\" (UID: \"d1c1a2b1-f312-47b1-b437-2d8f1760eea0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-98fnd" Oct 30 21:47:03.069400 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:03.069376 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-98fnd\" (UID: \"d1c1a2b1-f312-47b1-b437-2d8f1760eea0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-98fnd" Oct 30 21:47:03.069481 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:03.069417 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-98fnd\" (UID: \"d1c1a2b1-f312-47b1-b437-2d8f1760eea0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-98fnd" Oct 30 21:47:03.069549 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:03.069530 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-98fnd\" (UID: \"d1c1a2b1-f312-47b1-b437-2d8f1760eea0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-98fnd" Oct 30 21:47:03.071533 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:03.071505 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-98fnd"] Oct 30 21:47:03.171312 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:03.171277 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-98fnd\" (UID: \"d1c1a2b1-f312-47b1-b437-2d8f1760eea0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-98fnd" Oct 30 21:47:03.171490 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:03.171379 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-98fnd\" (UID: \"d1c1a2b1-f312-47b1-b437-2d8f1760eea0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-98fnd" Oct 30 21:47:03.171490 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:03.171420 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lv7l\" (UniqueName: \"kubernetes.io/projected/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-kube-api-access-6lv7l\") pod \"maas-default-gateway-openshift-default-75d7c785f-98fnd\" (UID: \"d1c1a2b1-f312-47b1-b437-2d8f1760eea0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-98fnd" Oct 30 21:47:03.171641 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:03.171493 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-98fnd\" (UID: \"d1c1a2b1-f312-47b1-b437-2d8f1760eea0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-98fnd" Oct 30 21:47:03.171641 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:03.171582 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-98fnd\" (UID: \"d1c1a2b1-f312-47b1-b437-2d8f1760eea0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-98fnd" Oct 30 21:47:03.171641 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:03.171625 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-98fnd\" (UID: \"d1c1a2b1-f312-47b1-b437-2d8f1760eea0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-98fnd" Oct 30 21:47:03.171812 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:03.171706 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-98fnd\" (UID: \"d1c1a2b1-f312-47b1-b437-2d8f1760eea0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-98fnd" Oct 30 21:47:03.171812 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:03.171785 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-98fnd\" (UID: \"d1c1a2b1-f312-47b1-b437-2d8f1760eea0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-98fnd" Oct 30 21:47:03.171949 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:03.171821 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-98fnd\" (UID: \"d1c1a2b1-f312-47b1-b437-2d8f1760eea0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-98fnd" Oct 30 21:47:03.171949 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:03.171858 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-98fnd\" (UID: \"d1c1a2b1-f312-47b1-b437-2d8f1760eea0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-98fnd" Oct 30 21:47:03.171949 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:03.171928 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-98fnd\" (UID: \"d1c1a2b1-f312-47b1-b437-2d8f1760eea0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-98fnd" Oct 30 21:47:03.172285 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:03.172253 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-98fnd\" (UID: \"d1c1a2b1-f312-47b1-b437-2d8f1760eea0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-98fnd" Oct 30 21:47:03.172420 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:03.172389 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-98fnd\" (UID: \"d1c1a2b1-f312-47b1-b437-2d8f1760eea0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-98fnd" Oct 30 21:47:03.172855 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:03.172823 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-98fnd\" (UID: \"d1c1a2b1-f312-47b1-b437-2d8f1760eea0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-98fnd" Oct 30 21:47:03.174252 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:03.174230 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-98fnd\" (UID: \"d1c1a2b1-f312-47b1-b437-2d8f1760eea0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-98fnd" Oct 30 21:47:03.174547 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:03.174528 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-98fnd\" (UID: \"d1c1a2b1-f312-47b1-b437-2d8f1760eea0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-98fnd" Oct 30 21:47:03.232829 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:03.232759 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-98fnd\" (UID: \"d1c1a2b1-f312-47b1-b437-2d8f1760eea0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-98fnd" Oct 30 21:47:03.232971 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:03.232940 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lv7l\" (UniqueName: \"kubernetes.io/projected/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-kube-api-access-6lv7l\") pod \"maas-default-gateway-openshift-default-75d7c785f-98fnd\" (UID: \"d1c1a2b1-f312-47b1-b437-2d8f1760eea0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-98fnd" Oct 30 21:47:03.276194 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:03.276166 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-98fnd" Oct 30 21:47:03.719115 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:47:03.719087 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1c1a2b1_f312_47b1_b437_2d8f1760eea0.slice/crio-31e3196705f7a08f4eeba9bbb51936f872896eda43a21afbf3346d580c2fcb74 WatchSource:0}: Error finding container 31e3196705f7a08f4eeba9bbb51936f872896eda43a21afbf3346d580c2fcb74: Status 404 returned error can't find the container with id 31e3196705f7a08f4eeba9bbb51936f872896eda43a21afbf3346d580c2fcb74 Oct 30 21:47:03.720528 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:03.720507 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-98fnd"] Oct 30 21:47:03.722073 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:03.722042 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:47:03.722179 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:03.722116 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:47:03.722179 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:03.722161 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:47:03.938881 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:03.938844 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-98fnd" event={"ID":"d1c1a2b1-f312-47b1-b437-2d8f1760eea0","Type":"ContainerStarted","Data":"a01f6e2e7773fa3f06e4d5a14a79b1ecb65ff4655a070fd2bd7e3357681df9f1"} Oct 30 21:47:03.939121 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:03.938927 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-98fnd" event={"ID":"d1c1a2b1-f312-47b1-b437-2d8f1760eea0","Type":"ContainerStarted","Data":"31e3196705f7a08f4eeba9bbb51936f872896eda43a21afbf3346d580c2fcb74"} Oct 30 21:47:04.065578 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:04.065484 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-98fnd" podStartSLOduration=2.065470949 podStartE2EDuration="2.065470949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:04.060194746 +0000 UTC m=+1407.802715682" watchObservedRunningTime="2026-10-30 21:47:04.065470949 +0000 UTC m=+1407.807991962" Oct 30 21:47:04.281175 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:04.281124 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-98fnd" Oct 30 21:47:04.283092 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:04.283057 5748 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-98fnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 30 21:47:04.283231 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:04.283113 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-98fnd" podUID="d1c1a2b1-f312-47b1-b437-2d8f1760eea0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 30 21:47:05.277247 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:05.277210 5748 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-98fnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 30 21:47:05.277674 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:05.277272 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-98fnd" podUID="d1c1a2b1-f312-47b1-b437-2d8f1760eea0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 30 21:47:06.277442 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:06.277406 5748 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-98fnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 30 21:47:06.277836 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:06.277471 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-98fnd" podUID="d1c1a2b1-f312-47b1-b437-2d8f1760eea0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 30 21:47:07.277504 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:07.277467 5748 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-98fnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 30 21:47:07.278017 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:07.277535 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-98fnd" podUID="d1c1a2b1-f312-47b1-b437-2d8f1760eea0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 30 21:47:08.276499 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:08.276462 5748 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-98fnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 30 21:47:08.276668 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:08.276527 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-98fnd" podUID="d1c1a2b1-f312-47b1-b437-2d8f1760eea0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 30 21:47:09.277260 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:09.277226 5748 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-98fnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 30 21:47:09.277686 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:09.277295 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-98fnd" podUID="d1c1a2b1-f312-47b1-b437-2d8f1760eea0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 30 21:47:10.277101 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:10.277063 5748 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-98fnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 30 21:47:10.277289 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:10.277123 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-98fnd" podUID="d1c1a2b1-f312-47b1-b437-2d8f1760eea0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 30 21:47:11.277187 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:11.277140 5748 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-98fnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 30 21:47:11.277393 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:11.277206 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-98fnd" podUID="d1c1a2b1-f312-47b1-b437-2d8f1760eea0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 30 21:47:12.298717 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:12.298683 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-98fnd" Oct 30 21:47:12.299679 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:12.299653 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-98fnd" Oct 30 21:47:12.300356 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:12.300336 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-98fnd" Oct 30 21:47:30.146418 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:47:30.146376 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:47:40.914441 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:40.914395 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c2588bc-xjvfc"] Oct 30 21:47:40.919297 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:40.919278 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2588bc-xjvfc" Oct 30 21:47:40.979551 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:40.979524 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c2588bc-xjvfc"] Oct 30 21:47:41.022363 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:41.022330 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:47:41.022587 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:41.022424 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:47:41.022587 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:41.022465 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:47:41.022771 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:41.022650 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 21:47:41.069876 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:41.069840 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa3a67e9-20ac-403a-981e-f426ffd13195-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c2588bc-xjvfc\" (UID: \"aa3a67e9-20ac-403a-981e-f426ffd13195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2588bc-xjvfc" Oct 30 21:47:41.070066 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:41.070029 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa3a67e9-20ac-403a-981e-f426ffd13195-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c2588bc-xjvfc\" (UID: \"aa3a67e9-20ac-403a-981e-f426ffd13195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2588bc-xjvfc" Oct 30 21:47:41.070169 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:41.070152 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79b2c\" (UniqueName: \"kubernetes.io/projected/aa3a67e9-20ac-403a-981e-f426ffd13195-kube-api-access-79b2c\") pod \"maas-api-revoke-sub-5c2588bc-xjvfc\" (UID: \"aa3a67e9-20ac-403a-981e-f426ffd13195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2588bc-xjvfc" Oct 30 21:47:41.171523 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:41.171444 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79b2c\" (UniqueName: \"kubernetes.io/projected/aa3a67e9-20ac-403a-981e-f426ffd13195-kube-api-access-79b2c\") pod \"maas-api-revoke-sub-5c2588bc-xjvfc\" (UID: \"aa3a67e9-20ac-403a-981e-f426ffd13195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2588bc-xjvfc" Oct 30 21:47:41.171681 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:41.171608 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa3a67e9-20ac-403a-981e-f426ffd13195-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c2588bc-xjvfc\" (UID: \"aa3a67e9-20ac-403a-981e-f426ffd13195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2588bc-xjvfc" Oct 30 21:47:41.171762 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:41.171735 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa3a67e9-20ac-403a-981e-f426ffd13195-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c2588bc-xjvfc\" (UID: \"aa3a67e9-20ac-403a-981e-f426ffd13195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2588bc-xjvfc" Oct 30 21:47:41.172406 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:41.172378 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa3a67e9-20ac-403a-981e-f426ffd13195-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c2588bc-xjvfc\" (UID: \"aa3a67e9-20ac-403a-981e-f426ffd13195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2588bc-xjvfc" Oct 30 21:47:41.174143 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:41.174124 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa3a67e9-20ac-403a-981e-f426ffd13195-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c2588bc-xjvfc\" (UID: \"aa3a67e9-20ac-403a-981e-f426ffd13195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2588bc-xjvfc" Oct 30 21:47:41.233955 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:41.233886 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79b2c\" (UniqueName: \"kubernetes.io/projected/aa3a67e9-20ac-403a-981e-f426ffd13195-kube-api-access-79b2c\") pod \"maas-api-revoke-sub-5c2588bc-xjvfc\" (UID: \"aa3a67e9-20ac-403a-981e-f426ffd13195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2588bc-xjvfc" Oct 30 21:47:41.533688 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:41.533612 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2588bc-xjvfc" Oct 30 21:47:41.917699 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:47:41.917664 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa3a67e9_20ac_403a_981e_f426ffd13195.slice/crio-8ea235e54eb085e38e9993a294e4ba3e42de4a9609688929ece81e4eb5843c4a WatchSource:0}: Error finding container 8ea235e54eb085e38e9993a294e4ba3e42de4a9609688929ece81e4eb5843c4a: Status 404 returned error can't find the container with id 8ea235e54eb085e38e9993a294e4ba3e42de4a9609688929ece81e4eb5843c4a Oct 30 21:47:41.918799 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:41.918774 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c2588bc-xjvfc"] Oct 30 21:47:42.151042 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:42.151008 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d456a94-rq8rs"] Oct 30 21:47:42.157647 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:42.157623 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d456a94-rq8rs" Oct 30 21:47:42.211138 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:42.211107 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d456a94-rq8rs"] Oct 30 21:47:42.231270 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:42.231240 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2588bc-xjvfc" event={"ID":"aa3a67e9-20ac-403a-981e-f426ffd13195","Type":"ContainerStarted","Data":"8ea235e54eb085e38e9993a294e4ba3e42de4a9609688929ece81e4eb5843c4a"} Oct 30 21:47:42.288465 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:42.288431 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49c25764-2e89-4473-841d-955cd2104467-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1d456a94-rq8rs\" (UID: \"49c25764-2e89-4473-841d-955cd2104467\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d456a94-rq8rs" Oct 30 21:47:42.288637 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:42.288542 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7896\" (UniqueName: \"kubernetes.io/projected/49c25764-2e89-4473-841d-955cd2104467-kube-api-access-g7896\") pod \"maas-api-revoke-keys-e2e-ait-1d456a94-rq8rs\" (UID: \"49c25764-2e89-4473-841d-955cd2104467\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d456a94-rq8rs" Oct 30 21:47:42.288637 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:42.288611 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49c25764-2e89-4473-841d-955cd2104467-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1d456a94-rq8rs\" (UID: \"49c25764-2e89-4473-841d-955cd2104467\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d456a94-rq8rs" Oct 30 21:47:42.389661 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:42.389628 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49c25764-2e89-4473-841d-955cd2104467-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1d456a94-rq8rs\" (UID: \"49c25764-2e89-4473-841d-955cd2104467\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d456a94-rq8rs" Oct 30 21:47:42.389826 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:42.389718 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7896\" (UniqueName: \"kubernetes.io/projected/49c25764-2e89-4473-841d-955cd2104467-kube-api-access-g7896\") pod \"maas-api-revoke-keys-e2e-ait-1d456a94-rq8rs\" (UID: \"49c25764-2e89-4473-841d-955cd2104467\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d456a94-rq8rs" Oct 30 21:47:42.389826 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:42.389792 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49c25764-2e89-4473-841d-955cd2104467-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1d456a94-rq8rs\" (UID: \"49c25764-2e89-4473-841d-955cd2104467\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d456a94-rq8rs" Oct 30 21:47:42.390354 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:42.390334 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49c25764-2e89-4473-841d-955cd2104467-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1d456a94-rq8rs\" (UID: \"49c25764-2e89-4473-841d-955cd2104467\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d456a94-rq8rs" Oct 30 21:47:42.392188 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:42.392159 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49c25764-2e89-4473-841d-955cd2104467-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1d456a94-rq8rs\" (UID: \"49c25764-2e89-4473-841d-955cd2104467\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d456a94-rq8rs" Oct 30 21:47:42.452126 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:42.452072 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7896\" (UniqueName: \"kubernetes.io/projected/49c25764-2e89-4473-841d-955cd2104467-kube-api-access-g7896\") pod \"maas-api-revoke-keys-e2e-ait-1d456a94-rq8rs\" (UID: \"49c25764-2e89-4473-841d-955cd2104467\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d456a94-rq8rs" Oct 30 21:47:42.472450 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:42.472427 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d456a94-rq8rs" Oct 30 21:47:42.861507 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:42.861478 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d456a94-rq8rs"] Oct 30 21:47:42.861630 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:47:42.861599 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49c25764_2e89_4473_841d_955cd2104467.slice/crio-d8cb79fcfcafc8b3519790baf87916ca5aa1104f9436c1c6bb841b15d015f13a WatchSource:0}: Error finding container d8cb79fcfcafc8b3519790baf87916ca5aa1104f9436c1c6bb841b15d015f13a: Status 404 returned error can't find the container with id d8cb79fcfcafc8b3519790baf87916ca5aa1104f9436c1c6bb841b15d015f13a Oct 30 21:47:43.241964 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:43.241882 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d456a94-rq8rs" event={"ID":"49c25764-2e89-4473-841d-955cd2104467","Type":"ContainerStarted","Data":"5939d8046c041a2205e70b77341d4eaaf8deefbdcfa135756a890253f2515a4d"} Oct 30 21:47:43.242403 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:43.242023 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d456a94-rq8rs" event={"ID":"49c25764-2e89-4473-841d-955cd2104467","Type":"ContainerStarted","Data":"d8cb79fcfcafc8b3519790baf87916ca5aa1104f9436c1c6bb841b15d015f13a"} Oct 30 21:47:43.243682 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:43.243651 5748 generic.go:356] "Generic (PLEG): container finished" podID="aa3a67e9-20ac-403a-981e-f426ffd13195" containerID="8a52594d3977ba0af3dca9271c4e34e7ce265ff7538f7da7d18c752c3a2884c9" exitCode=0 Oct 30 21:47:43.243821 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:43.243728 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2588bc-xjvfc" event={"ID":"aa3a67e9-20ac-403a-981e-f426ffd13195","Type":"ContainerDied","Data":"8a52594d3977ba0af3dca9271c4e34e7ce265ff7538f7da7d18c752c3a2884c9"} Oct 30 21:47:43.365664 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:43.365571 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d456a94-rq8rs" podStartSLOduration=1.365557938 podStartE2EDuration="1.365557938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:43.3650711 +0000 UTC m=+1447.107592038" watchObservedRunningTime="2026-10-30 21:47:43.365557938 +0000 UTC m=+1447.108078907" Oct 30 21:47:44.253722 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:44.253690 5748 generic.go:356] "Generic (PLEG): container finished" podID="49c25764-2e89-4473-841d-955cd2104467" containerID="5939d8046c041a2205e70b77341d4eaaf8deefbdcfa135756a890253f2515a4d" exitCode=0 Oct 30 21:47:44.254187 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:44.253781 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d456a94-rq8rs" event={"ID":"49c25764-2e89-4473-841d-955cd2104467","Type":"ContainerDied","Data":"5939d8046c041a2205e70b77341d4eaaf8deefbdcfa135756a890253f2515a4d"} Oct 30 21:47:44.422361 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:44.422332 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2588bc-xjvfc" Oct 30 21:47:44.522043 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:44.521967 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa3a67e9-20ac-403a-981e-f426ffd13195-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa3a67e9-20ac-403a-981e-f426ffd13195-maas-api-openshift-ca\") pod \"aa3a67e9-20ac-403a-981e-f426ffd13195\" (UID: \"aa3a67e9-20ac-403a-981e-f426ffd13195\") " Oct 30 21:47:44.522043 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:44.522020 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa3a67e9-20ac-403a-981e-f426ffd13195-kube-api-access-79b2c\" (UniqueName: \"kubernetes.io/projected/aa3a67e9-20ac-403a-981e-f426ffd13195-kube-api-access-79b2c\") pod \"aa3a67e9-20ac-403a-981e-f426ffd13195\" (UID: \"aa3a67e9-20ac-403a-981e-f426ffd13195\") " Oct 30 21:47:44.522259 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:44.522073 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa3a67e9-20ac-403a-981e-f426ffd13195-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa3a67e9-20ac-403a-981e-f426ffd13195-maas-api-xks-ca\") pod \"aa3a67e9-20ac-403a-981e-f426ffd13195\" (UID: \"aa3a67e9-20ac-403a-981e-f426ffd13195\") " Oct 30 21:47:44.522375 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:44.522351 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa3a67e9-20ac-403a-981e-f426ffd13195-maas-api-openshift-ca" pod "aa3a67e9-20ac-403a-981e-f426ffd13195" (UID: "aa3a67e9-20ac-403a-981e-f426ffd13195"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:47:44.522741 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:44.522722 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa3a67e9-20ac-403a-981e-f426ffd13195-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:44.524132 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:44.524108 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa3a67e9-20ac-403a-981e-f426ffd13195-kube-api-access-79b2c" pod "aa3a67e9-20ac-403a-981e-f426ffd13195" (UID: "aa3a67e9-20ac-403a-981e-f426ffd13195"). InnerVolumeSpecName "kube-api-access-79b2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:44.524132 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:44.524117 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa3a67e9-20ac-403a-981e-f426ffd13195-maas-api-xks-ca" pod "aa3a67e9-20ac-403a-981e-f426ffd13195" (UID: "aa3a67e9-20ac-403a-981e-f426ffd13195"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:47:44.624625 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:44.624589 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79b2c\" (UniqueName: \"kubernetes.io/projected/aa3a67e9-20ac-403a-981e-f426ffd13195-kube-api-access-79b2c\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:44.624625 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:44.624619 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa3a67e9-20ac-403a-981e-f426ffd13195-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:45.263544 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:45.263514 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2588bc-xjvfc" Oct 30 21:47:45.263544 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:45.263542 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2588bc-xjvfc" event={"ID":"aa3a67e9-20ac-403a-981e-f426ffd13195","Type":"ContainerDied","Data":"8ea235e54eb085e38e9993a294e4ba3e42de4a9609688929ece81e4eb5843c4a"} Oct 30 21:47:45.264018 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:45.263593 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ea235e54eb085e38e9993a294e4ba3e42de4a9609688929ece81e4eb5843c4a" Oct 30 21:47:45.428566 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:45.428543 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d456a94-rq8rs" Oct 30 21:47:45.536751 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:45.536671 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49c25764-2e89-4473-841d-955cd2104467-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49c25764-2e89-4473-841d-955cd2104467-maas-api-openshift-ca\") pod \"49c25764-2e89-4473-841d-955cd2104467\" (UID: \"49c25764-2e89-4473-841d-955cd2104467\") " Oct 30 21:47:45.536924 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:45.536851 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49c25764-2e89-4473-841d-955cd2104467-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49c25764-2e89-4473-841d-955cd2104467-maas-api-xks-ca\") pod \"49c25764-2e89-4473-841d-955cd2104467\" (UID: \"49c25764-2e89-4473-841d-955cd2104467\") " Oct 30 21:47:45.536983 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:45.536929 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49c25764-2e89-4473-841d-955cd2104467-kube-api-access-g7896\" (UniqueName: \"kubernetes.io/projected/49c25764-2e89-4473-841d-955cd2104467-kube-api-access-g7896\") pod \"49c25764-2e89-4473-841d-955cd2104467\" (UID: \"49c25764-2e89-4473-841d-955cd2104467\") " Oct 30 21:47:45.537109 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:45.537083 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49c25764-2e89-4473-841d-955cd2104467-maas-api-openshift-ca" pod "49c25764-2e89-4473-841d-955cd2104467" (UID: "49c25764-2e89-4473-841d-955cd2104467"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:47:45.537668 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:45.537645 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49c25764-2e89-4473-841d-955cd2104467-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:45.539022 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:45.538999 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49c25764-2e89-4473-841d-955cd2104467-maas-api-xks-ca" pod "49c25764-2e89-4473-841d-955cd2104467" (UID: "49c25764-2e89-4473-841d-955cd2104467"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:47:45.539154 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:45.539138 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49c25764-2e89-4473-841d-955cd2104467-kube-api-access-g7896" pod "49c25764-2e89-4473-841d-955cd2104467" (UID: "49c25764-2e89-4473-841d-955cd2104467"). InnerVolumeSpecName "kube-api-access-g7896". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:45.639420 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:45.639388 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49c25764-2e89-4473-841d-955cd2104467-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:45.639420 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:45.639416 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7896\" (UniqueName: \"kubernetes.io/projected/49c25764-2e89-4473-841d-955cd2104467-kube-api-access-g7896\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:46.272972 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:46.272942 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d456a94-rq8rs" Oct 30 21:47:46.273363 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:46.272973 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d456a94-rq8rs" event={"ID":"49c25764-2e89-4473-841d-955cd2104467","Type":"ContainerDied","Data":"d8cb79fcfcafc8b3519790baf87916ca5aa1104f9436c1c6bb841b15d015f13a"} Oct 30 21:47:46.273363 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:46.273034 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8cb79fcfcafc8b3519790baf87916ca5aa1104f9436c1c6bb841b15d015f13a" Oct 30 21:47:54.393342 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:54.393313 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e6f16377-fjbrr"] Oct 30 21:47:54.393943 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:54.393928 5748 cpu_manager.go:414] "containerMap: removing container" podUID="aa3a67e9-20ac-403a-981e-f426ffd13195" containerName="revoke-keys" Oct 30 21:47:54.393997 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:54.393944 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa3a67e9-20ac-403a-981e-f426ffd13195" containerName="revoke-keys" Oct 30 21:47:54.393997 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:54.393983 5748 cpu_manager.go:414] "containerMap: removing container" podUID="49c25764-2e89-4473-841d-955cd2104467" containerName="revoke-keys" Oct 30 21:47:54.393997 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:54.393989 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49c25764-2e89-4473-841d-955cd2104467" containerName="revoke-keys" Oct 30 21:47:54.394116 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:54.394107 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa3a67e9-20ac-403a-981e-f426ffd13195" containerName="revoke-keys" Oct 30 21:47:54.394149 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:54.394125 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="49c25764-2e89-4473-841d-955cd2104467" containerName="revoke-keys" Oct 30 21:47:54.397638 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:54.397619 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e6f16377-fjbrr" Oct 30 21:47:54.468942 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:54.465063 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qh5b\" (UniqueName: \"kubernetes.io/projected/ec82dbc6-0107-44f7-a5cd-2217a55af188-kube-api-access-4qh5b\") pod \"maas-api-revoke-keys-e2e-ait-e6f16377-fjbrr\" (UID: \"ec82dbc6-0107-44f7-a5cd-2217a55af188\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e6f16377-fjbrr" Oct 30 21:47:54.468942 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:54.465122 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec82dbc6-0107-44f7-a5cd-2217a55af188-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e6f16377-fjbrr\" (UID: \"ec82dbc6-0107-44f7-a5cd-2217a55af188\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e6f16377-fjbrr" Oct 30 21:47:54.468942 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:54.465186 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec82dbc6-0107-44f7-a5cd-2217a55af188-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e6f16377-fjbrr\" (UID: \"ec82dbc6-0107-44f7-a5cd-2217a55af188\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e6f16377-fjbrr" Oct 30 21:47:54.501833 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:54.501801 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:47:54.501833 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:54.501807 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:47:54.502090 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:54.501935 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:47:54.502090 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:54.501969 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 21:47:54.506844 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:54.506821 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e6f16377-fjbrr"] Oct 30 21:47:54.567137 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:54.567108 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qh5b\" (UniqueName: \"kubernetes.io/projected/ec82dbc6-0107-44f7-a5cd-2217a55af188-kube-api-access-4qh5b\") pod \"maas-api-revoke-keys-e2e-ait-e6f16377-fjbrr\" (UID: \"ec82dbc6-0107-44f7-a5cd-2217a55af188\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e6f16377-fjbrr" Oct 30 21:47:54.567254 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:54.567147 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec82dbc6-0107-44f7-a5cd-2217a55af188-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e6f16377-fjbrr\" (UID: \"ec82dbc6-0107-44f7-a5cd-2217a55af188\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e6f16377-fjbrr" Oct 30 21:47:54.567254 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:54.567200 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec82dbc6-0107-44f7-a5cd-2217a55af188-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e6f16377-fjbrr\" (UID: \"ec82dbc6-0107-44f7-a5cd-2217a55af188\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e6f16377-fjbrr" Oct 30 21:47:54.567869 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:54.567849 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec82dbc6-0107-44f7-a5cd-2217a55af188-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e6f16377-fjbrr\" (UID: \"ec82dbc6-0107-44f7-a5cd-2217a55af188\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e6f16377-fjbrr" Oct 30 21:47:54.569551 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:54.569533 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec82dbc6-0107-44f7-a5cd-2217a55af188-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e6f16377-fjbrr\" (UID: \"ec82dbc6-0107-44f7-a5cd-2217a55af188\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e6f16377-fjbrr" Oct 30 21:47:54.628620 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:54.628599 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qh5b\" (UniqueName: \"kubernetes.io/projected/ec82dbc6-0107-44f7-a5cd-2217a55af188-kube-api-access-4qh5b\") pod \"maas-api-revoke-keys-e2e-ait-e6f16377-fjbrr\" (UID: \"ec82dbc6-0107-44f7-a5cd-2217a55af188\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e6f16377-fjbrr" Oct 30 21:47:54.712243 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:54.712186 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e6f16377-fjbrr" Oct 30 21:47:55.080183 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:55.080093 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1225cc53-6hbt5"] Oct 30 21:47:55.085209 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:55.085181 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1225cc53-6hbt5" Oct 30 21:47:55.096616 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:47:55.096589 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec82dbc6_0107_44f7_a5cd_2217a55af188.slice/crio-7ea588dc1a5b4a4d7d6ae94cc678e963c98d26efbcce5fbfd10d2efa674b1088 WatchSource:0}: Error finding container 7ea588dc1a5b4a4d7d6ae94cc678e963c98d26efbcce5fbfd10d2efa674b1088: Status 404 returned error can't find the container with id 7ea588dc1a5b4a4d7d6ae94cc678e963c98d26efbcce5fbfd10d2efa674b1088 Oct 30 21:47:55.097560 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:55.097530 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e6f16377-fjbrr"] Oct 30 21:47:55.143248 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:55.143216 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1225cc53-6hbt5"] Oct 30 21:47:55.179514 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:55.179486 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fgjq\" (UniqueName: \"kubernetes.io/projected/d3ade189-3186-4a67-887f-c421d662d7f7-kube-api-access-2fgjq\") pod \"maas-api-revoke-keys-e2e-ait-1225cc53-6hbt5\" (UID: \"d3ade189-3186-4a67-887f-c421d662d7f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1225cc53-6hbt5" Oct 30 21:47:55.179654 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:55.179617 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3ade189-3186-4a67-887f-c421d662d7f7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1225cc53-6hbt5\" (UID: \"d3ade189-3186-4a67-887f-c421d662d7f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1225cc53-6hbt5" Oct 30 21:47:55.179771 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:55.179751 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3ade189-3186-4a67-887f-c421d662d7f7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1225cc53-6hbt5\" (UID: \"d3ade189-3186-4a67-887f-c421d662d7f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1225cc53-6hbt5" Oct 30 21:47:55.281243 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:55.281207 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3ade189-3186-4a67-887f-c421d662d7f7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1225cc53-6hbt5\" (UID: \"d3ade189-3186-4a67-887f-c421d662d7f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1225cc53-6hbt5" Oct 30 21:47:55.281416 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:55.281326 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fgjq\" (UniqueName: \"kubernetes.io/projected/d3ade189-3186-4a67-887f-c421d662d7f7-kube-api-access-2fgjq\") pod \"maas-api-revoke-keys-e2e-ait-1225cc53-6hbt5\" (UID: \"d3ade189-3186-4a67-887f-c421d662d7f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1225cc53-6hbt5" Oct 30 21:47:55.281478 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:55.281440 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3ade189-3186-4a67-887f-c421d662d7f7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1225cc53-6hbt5\" (UID: \"d3ade189-3186-4a67-887f-c421d662d7f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1225cc53-6hbt5" Oct 30 21:47:55.282209 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:55.282176 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3ade189-3186-4a67-887f-c421d662d7f7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1225cc53-6hbt5\" (UID: \"d3ade189-3186-4a67-887f-c421d662d7f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1225cc53-6hbt5" Oct 30 21:47:55.301654 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:55.301621 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3ade189-3186-4a67-887f-c421d662d7f7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1225cc53-6hbt5\" (UID: \"d3ade189-3186-4a67-887f-c421d662d7f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1225cc53-6hbt5" Oct 30 21:47:55.342476 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:55.342399 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e6f16377-fjbrr" event={"ID":"ec82dbc6-0107-44f7-a5cd-2217a55af188","Type":"ContainerStarted","Data":"7ea588dc1a5b4a4d7d6ae94cc678e963c98d26efbcce5fbfd10d2efa674b1088"} Oct 30 21:47:55.342939 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:55.342917 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fgjq\" (UniqueName: \"kubernetes.io/projected/d3ade189-3186-4a67-887f-c421d662d7f7-kube-api-access-2fgjq\") pod \"maas-api-revoke-keys-e2e-ait-1225cc53-6hbt5\" (UID: \"d3ade189-3186-4a67-887f-c421d662d7f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1225cc53-6hbt5" Oct 30 21:47:55.419183 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:55.416574 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1225cc53-6hbt5" Oct 30 21:47:55.847517 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:47:55.847486 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3ade189_3186_4a67_887f_c421d662d7f7.slice/crio-d690158179bbbd5de4ceefbb576bc97bc66f214ed2f9e9c57ad90a85124584ea WatchSource:0}: Error finding container d690158179bbbd5de4ceefbb576bc97bc66f214ed2f9e9c57ad90a85124584ea: Status 404 returned error can't find the container with id d690158179bbbd5de4ceefbb576bc97bc66f214ed2f9e9c57ad90a85124584ea Oct 30 21:47:55.847690 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:55.847667 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1225cc53-6hbt5"] Oct 30 21:47:56.351727 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:56.351694 5748 generic.go:356] "Generic (PLEG): container finished" podID="ec82dbc6-0107-44f7-a5cd-2217a55af188" containerID="8c6f26560cb5d014e8998ebe9c2cfdf8c33df29c9e5ff1342ed3e68db4f716e6" exitCode=0 Oct 30 21:47:56.351936 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:56.351777 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e6f16377-fjbrr" event={"ID":"ec82dbc6-0107-44f7-a5cd-2217a55af188","Type":"ContainerDied","Data":"8c6f26560cb5d014e8998ebe9c2cfdf8c33df29c9e5ff1342ed3e68db4f716e6"} Oct 30 21:47:56.353514 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:56.353489 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1225cc53-6hbt5" event={"ID":"d3ade189-3186-4a67-887f-c421d662d7f7","Type":"ContainerStarted","Data":"0909e3ac6b39bac974b51c83159a79980709ca5835787da976a920513f3a2608"} Oct 30 21:47:56.353625 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:56.353530 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1225cc53-6hbt5" event={"ID":"d3ade189-3186-4a67-887f-c421d662d7f7","Type":"ContainerStarted","Data":"d690158179bbbd5de4ceefbb576bc97bc66f214ed2f9e9c57ad90a85124584ea"} Oct 30 21:47:56.596414 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:56.596302 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1225cc53-6hbt5" podStartSLOduration=1.596283481 podStartE2EDuration="1.596283481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:56.595038907 +0000 UTC m=+1460.337559842" watchObservedRunningTime="2026-10-30 21:47:56.596283481 +0000 UTC m=+1460.338804455" Oct 30 21:47:57.363087 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:57.362998 5748 generic.go:356] "Generic (PLEG): container finished" podID="d3ade189-3186-4a67-887f-c421d662d7f7" containerID="0909e3ac6b39bac974b51c83159a79980709ca5835787da976a920513f3a2608" exitCode=0 Oct 30 21:47:57.363275 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:57.363082 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1225cc53-6hbt5" event={"ID":"d3ade189-3186-4a67-887f-c421d662d7f7","Type":"ContainerDied","Data":"0909e3ac6b39bac974b51c83159a79980709ca5835787da976a920513f3a2608"} Oct 30 21:47:57.548331 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:57.548307 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e6f16377-fjbrr" Oct 30 21:47:57.623073 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:57.622996 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec82dbc6-0107-44f7-a5cd-2217a55af188-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec82dbc6-0107-44f7-a5cd-2217a55af188-maas-api-openshift-ca\") pod \"ec82dbc6-0107-44f7-a5cd-2217a55af188\" (UID: \"ec82dbc6-0107-44f7-a5cd-2217a55af188\") " Oct 30 21:47:57.623451 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:57.623111 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec82dbc6-0107-44f7-a5cd-2217a55af188-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec82dbc6-0107-44f7-a5cd-2217a55af188-maas-api-xks-ca\") pod \"ec82dbc6-0107-44f7-a5cd-2217a55af188\" (UID: \"ec82dbc6-0107-44f7-a5cd-2217a55af188\") " Oct 30 21:47:57.623451 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:57.623152 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec82dbc6-0107-44f7-a5cd-2217a55af188-kube-api-access-4qh5b\" (UniqueName: \"kubernetes.io/projected/ec82dbc6-0107-44f7-a5cd-2217a55af188-kube-api-access-4qh5b\") pod \"ec82dbc6-0107-44f7-a5cd-2217a55af188\" (UID: \"ec82dbc6-0107-44f7-a5cd-2217a55af188\") " Oct 30 21:47:57.623451 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:57.623353 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec82dbc6-0107-44f7-a5cd-2217a55af188-maas-api-openshift-ca" pod "ec82dbc6-0107-44f7-a5cd-2217a55af188" (UID: "ec82dbc6-0107-44f7-a5cd-2217a55af188"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:47:57.623784 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:57.623766 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec82dbc6-0107-44f7-a5cd-2217a55af188-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:57.625156 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:57.625133 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec82dbc6-0107-44f7-a5cd-2217a55af188-maas-api-xks-ca" pod "ec82dbc6-0107-44f7-a5cd-2217a55af188" (UID: "ec82dbc6-0107-44f7-a5cd-2217a55af188"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:47:57.625311 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:57.625294 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec82dbc6-0107-44f7-a5cd-2217a55af188-kube-api-access-4qh5b" pod "ec82dbc6-0107-44f7-a5cd-2217a55af188" (UID: "ec82dbc6-0107-44f7-a5cd-2217a55af188"). InnerVolumeSpecName "kube-api-access-4qh5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:57.724776 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:57.724742 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec82dbc6-0107-44f7-a5cd-2217a55af188-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:57.724951 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:57.724778 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qh5b\" (UniqueName: \"kubernetes.io/projected/ec82dbc6-0107-44f7-a5cd-2217a55af188-kube-api-access-4qh5b\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:58.372539 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:58.372509 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e6f16377-fjbrr" Oct 30 21:47:58.372741 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:58.372544 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e6f16377-fjbrr" event={"ID":"ec82dbc6-0107-44f7-a5cd-2217a55af188","Type":"ContainerDied","Data":"7ea588dc1a5b4a4d7d6ae94cc678e963c98d26efbcce5fbfd10d2efa674b1088"} Oct 30 21:47:58.372741 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:58.372620 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ea588dc1a5b4a4d7d6ae94cc678e963c98d26efbcce5fbfd10d2efa674b1088" Oct 30 21:47:58.525175 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:58.525154 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1225cc53-6hbt5" Oct 30 21:47:58.643623 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:58.643589 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3ade189-3186-4a67-887f-c421d662d7f7-kube-api-access-2fgjq\" (UniqueName: \"kubernetes.io/projected/d3ade189-3186-4a67-887f-c421d662d7f7-kube-api-access-2fgjq\") pod \"d3ade189-3186-4a67-887f-c421d662d7f7\" (UID: \"d3ade189-3186-4a67-887f-c421d662d7f7\") " Oct 30 21:47:58.644242 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:58.643647 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3ade189-3186-4a67-887f-c421d662d7f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3ade189-3186-4a67-887f-c421d662d7f7-maas-api-xks-ca\") pod \"d3ade189-3186-4a67-887f-c421d662d7f7\" (UID: \"d3ade189-3186-4a67-887f-c421d662d7f7\") " Oct 30 21:47:58.644242 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:58.643767 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3ade189-3186-4a67-887f-c421d662d7f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3ade189-3186-4a67-887f-c421d662d7f7-maas-api-openshift-ca\") pod \"d3ade189-3186-4a67-887f-c421d662d7f7\" (UID: \"d3ade189-3186-4a67-887f-c421d662d7f7\") " Oct 30 21:47:58.645136 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:58.644777 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3ade189-3186-4a67-887f-c421d662d7f7-maas-api-openshift-ca" pod "d3ade189-3186-4a67-887f-c421d662d7f7" (UID: "d3ade189-3186-4a67-887f-c421d662d7f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:47:58.647625 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:58.647595 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3ade189-3186-4a67-887f-c421d662d7f7-maas-api-xks-ca" pod "d3ade189-3186-4a67-887f-c421d662d7f7" (UID: "d3ade189-3186-4a67-887f-c421d662d7f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:47:58.647927 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:58.647879 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3ade189-3186-4a67-887f-c421d662d7f7-kube-api-access-2fgjq" pod "d3ade189-3186-4a67-887f-c421d662d7f7" (UID: "d3ade189-3186-4a67-887f-c421d662d7f7"). InnerVolumeSpecName "kube-api-access-2fgjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:58.745379 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:58.745347 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3ade189-3186-4a67-887f-c421d662d7f7-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:58.745379 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:58.745376 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3ade189-3186-4a67-887f-c421d662d7f7-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:58.745559 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:58.745385 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fgjq\" (UniqueName: \"kubernetes.io/projected/d3ade189-3186-4a67-887f-c421d662d7f7-kube-api-access-2fgjq\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:59.049843 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:59.049743 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92937c3f-nwdx8"] Oct 30 21:47:59.054157 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:59.054123 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92937c3f-nwdx8"] Oct 30 21:47:59.381680 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:59.381602 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1225cc53-6hbt5" Oct 30 21:47:59.381680 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:59.381632 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1225cc53-6hbt5" event={"ID":"d3ade189-3186-4a67-887f-c421d662d7f7","Type":"ContainerDied","Data":"d690158179bbbd5de4ceefbb576bc97bc66f214ed2f9e9c57ad90a85124584ea"} Oct 30 21:47:59.381858 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:47:59.381681 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d690158179bbbd5de4ceefbb576bc97bc66f214ed2f9e9c57ad90a85124584ea" Oct 30 21:48:00.146088 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:48:00.146053 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:48:00.423627 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:00.423547 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5171ada-77cf-4fc9-95b9-040f31458098" path="/var/lib/kubelet/pods/f5171ada-77cf-4fc9-95b9-040f31458098/volumes" Oct 30 21:48:30.146356 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:48:30.146321 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:48:30.248344 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:30.248307 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d456a94-rq8rs"] Oct 30 21:48:30.295671 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:30.295630 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d456a94-rq8rs"] Oct 30 21:48:30.419221 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:30.419137 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49c25764-2e89-4473-841d-955cd2104467" path="/var/lib/kubelet/pods/49c25764-2e89-4473-841d-955cd2104467/volumes" Oct 30 21:48:36.498737 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:36.498702 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e6f16377-fjbrr"] Oct 30 21:48:36.553727 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:36.553689 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e6f16377-fjbrr"] Oct 30 21:48:36.680455 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:36.680411 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j4v8r_3df036ff-1c00-4ce2-9e86-8acbeedf6c82/ovn-acl-logging/0.log" Oct 30 21:48:36.680664 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:36.680645 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j4v8r_3df036ff-1c00-4ce2-9e86-8acbeedf6c82/ovn-acl-logging/0.log" Oct 30 21:48:37.901299 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:37.901262 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1225cc53-6hbt5"] Oct 30 21:48:37.952500 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:37.952464 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1225cc53-6hbt5"] Oct 30 21:48:38.420616 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:38.420568 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3ade189-3186-4a67-887f-c421d662d7f7" path="/var/lib/kubelet/pods/d3ade189-3186-4a67-887f-c421d662d7f7/volumes" Oct 30 21:48:38.421205 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:38.421182 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec82dbc6-0107-44f7-a5cd-2217a55af188" path="/var/lib/kubelet/pods/ec82dbc6-0107-44f7-a5cd-2217a55af188/volumes" Oct 30 21:48:46.061186 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:46.061150 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e9a6a20-7nl6f"] Oct 30 21:48:46.111221 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:46.111184 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e9a6a20-7nl6f"] Oct 30 21:48:46.418842 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:46.418799 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="badf2444-5704-4625-83df-a827672f494e" path="/var/lib/kubelet/pods/badf2444-5704-4625-83df-a827672f494e/volumes" Oct 30 21:48:48.784402 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:48.784364 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bxj6r"] Oct 30 21:48:48.785070 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:48.785044 5748 cpu_manager.go:414] "containerMap: removing container" podUID="d3ade189-3186-4a67-887f-c421d662d7f7" containerName="revoke-keys" Oct 30 21:48:48.785070 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:48.785066 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3ade189-3186-4a67-887f-c421d662d7f7" containerName="revoke-keys" Oct 30 21:48:48.785286 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:48.785117 5748 cpu_manager.go:414] "containerMap: removing container" podUID="ec82dbc6-0107-44f7-a5cd-2217a55af188" containerName="revoke-keys" Oct 30 21:48:48.785286 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:48.785125 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec82dbc6-0107-44f7-a5cd-2217a55af188" containerName="revoke-keys" Oct 30 21:48:48.785395 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:48.785381 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3ade189-3186-4a67-887f-c421d662d7f7" containerName="revoke-keys" Oct 30 21:48:48.785446 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:48.785409 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec82dbc6-0107-44f7-a5cd-2217a55af188" containerName="revoke-keys" Oct 30 21:48:48.791010 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:48.790986 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bxj6r" Oct 30 21:48:48.893294 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:48.893257 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c04ccfd-188d-4566-bf40-2732890754d6-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-bxj6r\" (UID: \"6c04ccfd-188d-4566-bf40-2732890754d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bxj6r" Oct 30 21:48:48.893294 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:48.893290 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c04ccfd-188d-4566-bf40-2732890754d6-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-bxj6r\" (UID: \"6c04ccfd-188d-4566-bf40-2732890754d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bxj6r" Oct 30 21:48:48.893526 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:48.893473 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c04ccfd-188d-4566-bf40-2732890754d6-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-bxj6r\" (UID: \"6c04ccfd-188d-4566-bf40-2732890754d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bxj6r" Oct 30 21:48:48.893597 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:48.893525 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c04ccfd-188d-4566-bf40-2732890754d6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-bxj6r\" (UID: \"6c04ccfd-188d-4566-bf40-2732890754d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bxj6r" Oct 30 21:48:48.893662 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:48.893634 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c04ccfd-188d-4566-bf40-2732890754d6-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-bxj6r\" (UID: \"6c04ccfd-188d-4566-bf40-2732890754d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bxj6r" Oct 30 21:48:48.893719 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:48.893710 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c04ccfd-188d-4566-bf40-2732890754d6-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-bxj6r\" (UID: \"6c04ccfd-188d-4566-bf40-2732890754d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bxj6r" Oct 30 21:48:48.893799 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:48.893785 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c04ccfd-188d-4566-bf40-2732890754d6-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-bxj6r\" (UID: \"6c04ccfd-188d-4566-bf40-2732890754d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bxj6r" Oct 30 21:48:48.893860 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:48.893826 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c04ccfd-188d-4566-bf40-2732890754d6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-bxj6r\" (UID: \"6c04ccfd-188d-4566-bf40-2732890754d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bxj6r" Oct 30 21:48:48.893967 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:48.893942 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzs59\" (UniqueName: \"kubernetes.io/projected/6c04ccfd-188d-4566-bf40-2732890754d6-kube-api-access-kzs59\") pod \"maas-default-gateway-openshift-default-75d7c785f-bxj6r\" (UID: \"6c04ccfd-188d-4566-bf40-2732890754d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bxj6r" Oct 30 21:48:48.995256 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:48.995210 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c04ccfd-188d-4566-bf40-2732890754d6-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-bxj6r\" (UID: \"6c04ccfd-188d-4566-bf40-2732890754d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bxj6r" Oct 30 21:48:48.996066 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:48.996015 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c04ccfd-188d-4566-bf40-2732890754d6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-bxj6r\" (UID: \"6c04ccfd-188d-4566-bf40-2732890754d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bxj6r" Oct 30 21:48:48.996200 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:48.996074 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzs59\" (UniqueName: \"kubernetes.io/projected/6c04ccfd-188d-4566-bf40-2732890754d6-kube-api-access-kzs59\") pod \"maas-default-gateway-openshift-default-75d7c785f-bxj6r\" (UID: \"6c04ccfd-188d-4566-bf40-2732890754d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bxj6r" Oct 30 21:48:48.996293 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:48.996270 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c04ccfd-188d-4566-bf40-2732890754d6-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-bxj6r\" (UID: \"6c04ccfd-188d-4566-bf40-2732890754d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bxj6r" Oct 30 21:48:48.996353 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:48.996307 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c04ccfd-188d-4566-bf40-2732890754d6-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-bxj6r\" (UID: \"6c04ccfd-188d-4566-bf40-2732890754d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bxj6r" Oct 30 21:48:48.996567 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:48.996539 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c04ccfd-188d-4566-bf40-2732890754d6-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-bxj6r\" (UID: \"6c04ccfd-188d-4566-bf40-2732890754d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bxj6r" Oct 30 21:48:48.996693 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:48.996585 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c04ccfd-188d-4566-bf40-2732890754d6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-bxj6r\" (UID: \"6c04ccfd-188d-4566-bf40-2732890754d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bxj6r" Oct 30 21:48:48.996693 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:48.996630 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c04ccfd-188d-4566-bf40-2732890754d6-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-bxj6r\" (UID: \"6c04ccfd-188d-4566-bf40-2732890754d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bxj6r" Oct 30 21:48:48.996693 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:48.996683 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c04ccfd-188d-4566-bf40-2732890754d6-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-bxj6r\" (UID: \"6c04ccfd-188d-4566-bf40-2732890754d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bxj6r" Oct 30 21:48:48.997380 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:48.997174 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c04ccfd-188d-4566-bf40-2732890754d6-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-bxj6r\" (UID: \"6c04ccfd-188d-4566-bf40-2732890754d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bxj6r" Oct 30 21:48:48.997483 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:48.997425 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c04ccfd-188d-4566-bf40-2732890754d6-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-bxj6r\" (UID: \"6c04ccfd-188d-4566-bf40-2732890754d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bxj6r" Oct 30 21:48:48.997686 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:48.997663 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c04ccfd-188d-4566-bf40-2732890754d6-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-bxj6r\" (UID: \"6c04ccfd-188d-4566-bf40-2732890754d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bxj6r" Oct 30 21:48:48.997837 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:48.997795 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c04ccfd-188d-4566-bf40-2732890754d6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-bxj6r\" (UID: \"6c04ccfd-188d-4566-bf40-2732890754d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bxj6r" Oct 30 21:48:48.998255 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:48.998221 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c04ccfd-188d-4566-bf40-2732890754d6-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-bxj6r\" (UID: \"6c04ccfd-188d-4566-bf40-2732890754d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bxj6r" Oct 30 21:48:49.000683 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:49.000639 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c04ccfd-188d-4566-bf40-2732890754d6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-bxj6r\" (UID: \"6c04ccfd-188d-4566-bf40-2732890754d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bxj6r" Oct 30 21:48:49.005949 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:49.005924 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bxj6r"] Oct 30 21:48:49.008496 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:49.008472 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c04ccfd-188d-4566-bf40-2732890754d6-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-bxj6r\" (UID: \"6c04ccfd-188d-4566-bf40-2732890754d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bxj6r" Oct 30 21:48:49.058808 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:49.058734 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzs59\" (UniqueName: \"kubernetes.io/projected/6c04ccfd-188d-4566-bf40-2732890754d6-kube-api-access-kzs59\") pod \"maas-default-gateway-openshift-default-75d7c785f-bxj6r\" (UID: \"6c04ccfd-188d-4566-bf40-2732890754d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bxj6r" Oct 30 21:48:49.059359 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:49.059334 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c04ccfd-188d-4566-bf40-2732890754d6-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-bxj6r\" (UID: \"6c04ccfd-188d-4566-bf40-2732890754d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bxj6r" Oct 30 21:48:49.107151 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:49.107112 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bxj6r" Oct 30 21:48:49.579213 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:49.579174 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bxj6r"] Oct 30 21:48:49.579524 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:48:49.579496 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c04ccfd_188d_4566_bf40_2732890754d6.slice/crio-93759c149a13e31810fd7bb0cdff193b67d007083530ffb4f427cb2c509863ec WatchSource:0}: Error finding container 93759c149a13e31810fd7bb0cdff193b67d007083530ffb4f427cb2c509863ec: Status 404 returned error can't find the container with id 93759c149a13e31810fd7bb0cdff193b67d007083530ffb4f427cb2c509863ec Oct 30 21:48:49.582103 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:49.582073 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:48:49.582184 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:49.582134 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:48:49.582184 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:49.582164 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:48:49.995198 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:49.995155 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bxj6r" event={"ID":"6c04ccfd-188d-4566-bf40-2732890754d6","Type":"ContainerStarted","Data":"4dde1979ea38af11d2d62f454c53385cee62f20bcad15fb3c9a84b69fd4727cf"} Oct 30 21:48:49.996140 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:49.996104 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bxj6r" event={"ID":"6c04ccfd-188d-4566-bf40-2732890754d6","Type":"ContainerStarted","Data":"93759c149a13e31810fd7bb0cdff193b67d007083530ffb4f427cb2c509863ec"} Oct 30 21:48:50.108249 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:50.108211 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bxj6r" Oct 30 21:48:50.109807 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:50.109777 5748 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-bxj6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 21:48:50.109957 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:50.109837 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bxj6r" podUID="6c04ccfd-188d-4566-bf40-2732890754d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 21:48:50.217057 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:50.216999 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bxj6r" podStartSLOduration=2.21698452 podStartE2EDuration="2.21698452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:50.168020154 +0000 UTC m=+1513.910541089" watchObservedRunningTime="2026-10-30 21:48:50.21698452 +0000 UTC m=+1513.959505514" Oct 30 21:48:51.049507 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:51.049470 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0413d01a-65xkj"] Oct 30 21:48:51.099573 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:51.099538 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0413d01a-65xkj"] Oct 30 21:48:51.110388 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:51.110346 5748 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-bxj6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 21:48:51.110713 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:51.110678 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bxj6r" podUID="6c04ccfd-188d-4566-bf40-2732890754d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 21:48:52.124922 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:52.124340 5748 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-bxj6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 21:48:52.124922 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:52.124407 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bxj6r" podUID="6c04ccfd-188d-4566-bf40-2732890754d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 21:48:52.473562 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:52.473524 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62960998-49fb-43ff-aeb0-c1201517e2ab" path="/var/lib/kubelet/pods/62960998-49fb-43ff-aeb0-c1201517e2ab/volumes" Oct 30 21:48:53.111290 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:53.111249 5748 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-bxj6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 21:48:53.111615 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:53.111576 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bxj6r" podUID="6c04ccfd-188d-4566-bf40-2732890754d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 21:48:54.125109 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:54.125068 5748 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-bxj6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 21:48:54.125629 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:54.125128 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bxj6r" podUID="6c04ccfd-188d-4566-bf40-2732890754d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 21:48:55.051717 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:55.051681 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a51f0adb-x22tr"] Oct 30 21:48:55.120151 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:55.120108 5748 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-bxj6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 21:48:55.120363 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:55.120210 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bxj6r" podUID="6c04ccfd-188d-4566-bf40-2732890754d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 21:48:55.126570 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:55.126534 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a51f0adb-x22tr"] Oct 30 21:48:56.108186 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:56.108149 5748 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-bxj6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 21:48:56.108375 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:56.108218 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bxj6r" podUID="6c04ccfd-188d-4566-bf40-2732890754d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 21:48:56.422314 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:56.422278 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1a00bc7-6808-4176-b68c-c398ce86d586" path="/var/lib/kubelet/pods/f1a00bc7-6808-4176-b68c-c398ce86d586/volumes" Oct 30 21:48:57.107634 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:57.107600 5748 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-bxj6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 21:48:57.107842 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:57.107669 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bxj6r" podUID="6c04ccfd-188d-4566-bf40-2732890754d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 21:48:58.474065 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:58.474033 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bxj6r" Oct 30 21:48:58.474622 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:58.474470 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bxj6r" Oct 30 21:48:58.475288 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:48:58.475250 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bxj6r" Oct 30 21:49:00.145642 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:49:00.145596 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:49:06.056882 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:06.056844 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf2dd5b6-k54j2"] Oct 30 21:49:06.106851 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:06.106816 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf2dd5b6-k54j2"] Oct 30 21:49:06.418970 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:06.418885 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c568b44c-1112-4bea-befa-a58118e54637" path="/var/lib/kubelet/pods/c568b44c-1112-4bea-befa-a58118e54637/volumes" Oct 30 21:49:11.052282 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:11.052244 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd7fe367-chq4v"] Oct 30 21:49:11.102405 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:11.102367 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd7fe367-chq4v"] Oct 30 21:49:12.417811 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:12.417734 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d121da3d-7c63-4cf8-8fc6-7c87818ccc8c" path="/var/lib/kubelet/pods/d121da3d-7c63-4cf8-8fc6-7c87818ccc8c/volumes" Oct 30 21:49:12.816080 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:12.815991 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4259d052-phh64"] Oct 30 21:49:12.821662 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:12.821636 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4259d052-phh64" Oct 30 21:49:12.878881 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:12.878850 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4259d052-phh64"] Oct 30 21:49:12.907230 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:12.907196 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/889f25ea-9b78-4350-a0fb-770b4626a4e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4259d052-phh64\" (UID: \"889f25ea-9b78-4350-a0fb-770b4626a4e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4259d052-phh64" Oct 30 21:49:12.907392 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:12.907377 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/889f25ea-9b78-4350-a0fb-770b4626a4e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4259d052-phh64\" (UID: \"889f25ea-9b78-4350-a0fb-770b4626a4e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4259d052-phh64" Oct 30 21:49:12.907445 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:12.907408 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26f6z\" (UniqueName: \"kubernetes.io/projected/889f25ea-9b78-4350-a0fb-770b4626a4e6-kube-api-access-26f6z\") pod \"maas-api-revoke-sub-4259d052-phh64\" (UID: \"889f25ea-9b78-4350-a0fb-770b4626a4e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4259d052-phh64" Oct 30 21:49:12.924123 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:12.924096 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:49:12.924298 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:12.924142 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:49:12.924298 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:12.924185 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:49:12.925143 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:12.925122 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 21:49:13.009205 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:13.009166 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/889f25ea-9b78-4350-a0fb-770b4626a4e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4259d052-phh64\" (UID: \"889f25ea-9b78-4350-a0fb-770b4626a4e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4259d052-phh64" Oct 30 21:49:13.009205 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:13.009202 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26f6z\" (UniqueName: \"kubernetes.io/projected/889f25ea-9b78-4350-a0fb-770b4626a4e6-kube-api-access-26f6z\") pod \"maas-api-revoke-sub-4259d052-phh64\" (UID: \"889f25ea-9b78-4350-a0fb-770b4626a4e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4259d052-phh64" Oct 30 21:49:13.009416 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:13.009388 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/889f25ea-9b78-4350-a0fb-770b4626a4e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4259d052-phh64\" (UID: \"889f25ea-9b78-4350-a0fb-770b4626a4e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4259d052-phh64" Oct 30 21:49:13.010082 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:13.010057 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/889f25ea-9b78-4350-a0fb-770b4626a4e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4259d052-phh64\" (UID: \"889f25ea-9b78-4350-a0fb-770b4626a4e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4259d052-phh64" Oct 30 21:49:13.011589 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:13.011565 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/889f25ea-9b78-4350-a0fb-770b4626a4e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4259d052-phh64\" (UID: \"889f25ea-9b78-4350-a0fb-770b4626a4e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4259d052-phh64" Oct 30 21:49:13.071150 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:13.071084 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26f6z\" (UniqueName: \"kubernetes.io/projected/889f25ea-9b78-4350-a0fb-770b4626a4e6-kube-api-access-26f6z\") pod \"maas-api-revoke-sub-4259d052-phh64\" (UID: \"889f25ea-9b78-4350-a0fb-770b4626a4e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4259d052-phh64" Oct 30 21:49:13.138026 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:13.137993 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4259d052-phh64" Oct 30 21:49:13.522741 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:49:13.522708 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod889f25ea_9b78_4350_a0fb_770b4626a4e6.slice/crio-be201f1bc67cb023620022e355ee7d4a8004b873e2f96d082660113d11c5b80b WatchSource:0}: Error finding container be201f1bc67cb023620022e355ee7d4a8004b873e2f96d082660113d11c5b80b: Status 404 returned error can't find the container with id be201f1bc67cb023620022e355ee7d4a8004b873e2f96d082660113d11c5b80b Oct 30 21:49:13.523878 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:13.523850 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4259d052-phh64"] Oct 30 21:49:14.218198 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:14.218098 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4259d052-phh64" event={"ID":"889f25ea-9b78-4350-a0fb-770b4626a4e6","Type":"ContainerStarted","Data":"46b04822991af3fc25a0613fe30f493e9cf365996fd170509bfaa999e952276b"} Oct 30 21:49:14.218198 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:14.218173 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4259d052-phh64" event={"ID":"889f25ea-9b78-4350-a0fb-770b4626a4e6","Type":"ContainerStarted","Data":"be201f1bc67cb023620022e355ee7d4a8004b873e2f96d082660113d11c5b80b"} Oct 30 21:49:14.341040 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:14.340973 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4259d052-phh64" podStartSLOduration=2.340952155 podStartE2EDuration="2.340952155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:14.334992247 +0000 UTC m=+1538.077513185" watchObservedRunningTime="2026-10-30 21:49:14.340952155 +0000 UTC m=+1538.083473093" Oct 30 21:49:15.057920 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:15.054430 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0ea77a6-pf7qk"] Oct 30 21:49:15.105096 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:15.105059 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0ea77a6-pf7qk"] Oct 30 21:49:15.230120 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:15.230083 5748 generic.go:356] "Generic (PLEG): container finished" podID="889f25ea-9b78-4350-a0fb-770b4626a4e6" containerID="46b04822991af3fc25a0613fe30f493e9cf365996fd170509bfaa999e952276b" exitCode=0 Oct 30 21:49:15.230289 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:15.230169 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4259d052-phh64" event={"ID":"889f25ea-9b78-4350-a0fb-770b4626a4e6","Type":"ContainerDied","Data":"46b04822991af3fc25a0613fe30f493e9cf365996fd170509bfaa999e952276b"} Oct 30 21:49:16.416449 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:16.416371 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4259d052-phh64" Oct 30 21:49:16.419853 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:16.419823 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0749f3f-6b1e-4e78-9ab2-9d4c8b65e46e" path="/var/lib/kubelet/pods/b0749f3f-6b1e-4e78-9ab2-9d4c8b65e46e/volumes" Oct 30 21:49:16.479540 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:16.479492 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/889f25ea-9b78-4350-a0fb-770b4626a4e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/889f25ea-9b78-4350-a0fb-770b4626a4e6-maas-api-xks-ca\") pod \"889f25ea-9b78-4350-a0fb-770b4626a4e6\" (UID: \"889f25ea-9b78-4350-a0fb-770b4626a4e6\") " Oct 30 21:49:16.479730 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:16.479650 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/889f25ea-9b78-4350-a0fb-770b4626a4e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/889f25ea-9b78-4350-a0fb-770b4626a4e6-maas-api-openshift-ca\") pod \"889f25ea-9b78-4350-a0fb-770b4626a4e6\" (UID: \"889f25ea-9b78-4350-a0fb-770b4626a4e6\") " Oct 30 21:49:16.480114 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:16.480082 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/889f25ea-9b78-4350-a0fb-770b4626a4e6-maas-api-openshift-ca" pod "889f25ea-9b78-4350-a0fb-770b4626a4e6" (UID: "889f25ea-9b78-4350-a0fb-770b4626a4e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:16.480257 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:16.480242 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/889f25ea-9b78-4350-a0fb-770b4626a4e6-kube-api-access-26f6z\" (UniqueName: \"kubernetes.io/projected/889f25ea-9b78-4350-a0fb-770b4626a4e6-kube-api-access-26f6z\") pod \"889f25ea-9b78-4350-a0fb-770b4626a4e6\" (UID: \"889f25ea-9b78-4350-a0fb-770b4626a4e6\") " Oct 30 21:49:16.480936 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:16.480885 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/889f25ea-9b78-4350-a0fb-770b4626a4e6-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:16.482114 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:16.482085 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/889f25ea-9b78-4350-a0fb-770b4626a4e6-maas-api-xks-ca" pod "889f25ea-9b78-4350-a0fb-770b4626a4e6" (UID: "889f25ea-9b78-4350-a0fb-770b4626a4e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:49:16.482433 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:16.482407 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/889f25ea-9b78-4350-a0fb-770b4626a4e6-kube-api-access-26f6z" pod "889f25ea-9b78-4350-a0fb-770b4626a4e6" (UID: "889f25ea-9b78-4350-a0fb-770b4626a4e6"). InnerVolumeSpecName "kube-api-access-26f6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:16.582476 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:16.582443 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/889f25ea-9b78-4350-a0fb-770b4626a4e6-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:16.582476 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:16.582471 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26f6z\" (UniqueName: \"kubernetes.io/projected/889f25ea-9b78-4350-a0fb-770b4626a4e6-kube-api-access-26f6z\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:17.247767 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:17.247735 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4259d052-phh64" Oct 30 21:49:17.247953 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:17.247775 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4259d052-phh64" event={"ID":"889f25ea-9b78-4350-a0fb-770b4626a4e6","Type":"ContainerDied","Data":"be201f1bc67cb023620022e355ee7d4a8004b873e2f96d082660113d11c5b80b"} Oct 30 21:49:17.247953 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:17.247870 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be201f1bc67cb023620022e355ee7d4a8004b873e2f96d082660113d11c5b80b" Oct 30 21:49:28.048688 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:28.048652 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-030f7de6-4t9x2"] Oct 30 21:49:28.099104 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:28.099064 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-030f7de6-4t9x2"] Oct 30 21:49:28.417978 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:28.417945 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee3cb1c0-6bfc-4983-9be4-98e1f9b0b762" path="/var/lib/kubelet/pods/ee3cb1c0-6bfc-4983-9be4-98e1f9b0b762/volumes" Oct 30 21:49:30.146801 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:49:30.146762 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:49:31.050393 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:31.050349 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fd22cad-cskvk"] Oct 30 21:49:31.101164 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:31.101123 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fd22cad-cskvk"] Oct 30 21:49:32.417779 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:32.417738 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da88fa3f-81d6-4fef-b231-7641ca76ea88" path="/var/lib/kubelet/pods/da88fa3f-81d6-4fef-b231-7641ca76ea88/volumes" Oct 30 21:49:37.009020 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:37.008986 5748 scope.go:122] "RemoveContainer" containerID="1236039daad2d4cdc580ffc77b1c2ee82430557f25cf0c08030a4f889b7a16c4" Oct 30 21:49:44.066811 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:44.066766 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83bee69b-khpsw"] Oct 30 21:49:44.132921 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:44.131726 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83bee69b-khpsw"] Oct 30 21:49:44.442309 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:44.442269 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af33c160-83ca-4ac8-b2fd-4273ff2c0656" path="/var/lib/kubelet/pods/af33c160-83ca-4ac8-b2fd-4273ff2c0656/volumes" Oct 30 21:49:46.054568 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:46.054532 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8192d23-tm9fg"] Oct 30 21:49:46.105451 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:46.105363 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8192d23-tm9fg"] Oct 30 21:49:46.435429 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:46.435384 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50a27536-551f-47f3-8ae8-d754fadf7a71" path="/var/lib/kubelet/pods/50a27536-551f-47f3-8ae8-d754fadf7a71/volumes" Oct 30 21:49:56.054644 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:56.054607 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02b9f950-cq6mz"] Oct 30 21:49:56.105880 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:56.105848 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bc1bfa1-s9d7h"] Oct 30 21:49:56.111880 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:56.111857 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8babaae-hl6tw"] Oct 30 21:49:56.117208 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:56.117188 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02b9f950-cq6mz"] Oct 30 21:49:56.121163 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:56.121141 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8babaae-hl6tw"] Oct 30 21:49:56.154795 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:56.154774 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bc1bfa1-s9d7h"] Oct 30 21:49:56.421839 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:56.421801 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14682f40-c39a-4234-9d38-947c5db6201b" path="/var/lib/kubelet/pods/14682f40-c39a-4234-9d38-947c5db6201b/volumes" Oct 30 21:49:56.422426 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:56.422395 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="889613b9-6b8d-4ebc-a9ac-d9b2e9d8d817" path="/var/lib/kubelet/pods/889613b9-6b8d-4ebc-a9ac-d9b2e9d8d817/volumes" Oct 30 21:49:56.422909 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:49:56.422883 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4b02ecf-7f32-4555-a5b0-2821f21ad666" path="/var/lib/kubelet/pods/d4b02ecf-7f32-4555-a5b0-2821f21ad666/volumes" Oct 30 21:50:00.146589 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:50:00.146550 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:50:02.050881 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:02.050843 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d69ca2e6-dx6dj"] Oct 30 21:50:02.101261 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:02.101224 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d69ca2e6-dx6dj"] Oct 30 21:50:02.419062 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:02.419028 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe58c836-3dbd-4389-83c8-f66b58152106" path="/var/lib/kubelet/pods/fe58c836-3dbd-4389-83c8-f66b58152106/volumes" Oct 30 21:50:07.895468 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:07.895430 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q"] Oct 30 21:50:07.895950 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:07.895637 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q" podUID="00245d73-3768-4ecd-9ddb-371c69b05e56" containerName="istio-proxy" containerID="cri-o://c37162bb04d8c3387b63a6bde42678db1be3569fb39897e7d750a707127aaa38" gracePeriod=30 Oct 30 21:50:11.052092 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:11.052054 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dde10fdf-j8rq6"] Oct 30 21:50:11.111998 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:11.111949 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dde10fdf-j8rq6"] Oct 30 21:50:11.116880 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:11.116847 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-962109ad-8km6j"] Oct 30 21:50:11.120830 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:11.120802 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-962109ad-8km6j"] Oct 30 21:50:12.418752 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:12.418707 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c22b3be-2a12-46e0-8465-a319ddede7e4" path="/var/lib/kubelet/pods/6c22b3be-2a12-46e0-8465-a319ddede7e4/volumes" Oct 30 21:50:12.419434 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:12.419409 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fab2565b-8b9b-4e45-baa2-6ec1f5cf002f" path="/var/lib/kubelet/pods/fab2565b-8b9b-4e45-baa2-6ec1f5cf002f/volumes" Oct 30 21:50:13.228376 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:13.228351 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q" Oct 30 21:50:13.303279 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:13.303248 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00245d73-3768-4ecd-9ddb-371c69b05e56-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00245d73-3768-4ecd-9ddb-371c69b05e56-workload-certs\") pod \"00245d73-3768-4ecd-9ddb-371c69b05e56\" (UID: \"00245d73-3768-4ecd-9ddb-371c69b05e56\") " Oct 30 21:50:13.303453 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:13.303377 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00245d73-3768-4ecd-9ddb-371c69b05e56-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00245d73-3768-4ecd-9ddb-371c69b05e56-istiod-ca-cert\") pod \"00245d73-3768-4ecd-9ddb-371c69b05e56\" (UID: \"00245d73-3768-4ecd-9ddb-371c69b05e56\") " Oct 30 21:50:13.303453 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:13.303423 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/00245d73-3768-4ecd-9ddb-371c69b05e56-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00245d73-3768-4ecd-9ddb-371c69b05e56-istio-podinfo\") pod \"00245d73-3768-4ecd-9ddb-371c69b05e56\" (UID: \"00245d73-3768-4ecd-9ddb-371c69b05e56\") " Oct 30 21:50:13.303555 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:13.303469 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00245d73-3768-4ecd-9ddb-371c69b05e56-istio-token\" (UniqueName: \"kubernetes.io/projected/00245d73-3768-4ecd-9ddb-371c69b05e56-istio-token\") pod \"00245d73-3768-4ecd-9ddb-371c69b05e56\" (UID: \"00245d73-3768-4ecd-9ddb-371c69b05e56\") " Oct 30 21:50:13.303555 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:13.303501 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00245d73-3768-4ecd-9ddb-371c69b05e56-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00245d73-3768-4ecd-9ddb-371c69b05e56-workload-socket\") pod \"00245d73-3768-4ecd-9ddb-371c69b05e56\" (UID: \"00245d73-3768-4ecd-9ddb-371c69b05e56\") " Oct 30 21:50:13.303555 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:13.303552 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00245d73-3768-4ecd-9ddb-371c69b05e56-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00245d73-3768-4ecd-9ddb-371c69b05e56-istio-envoy\") pod \"00245d73-3768-4ecd-9ddb-371c69b05e56\" (UID: \"00245d73-3768-4ecd-9ddb-371c69b05e56\") " Oct 30 21:50:13.303729 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:13.303552 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00245d73-3768-4ecd-9ddb-371c69b05e56-workload-certs" pod "00245d73-3768-4ecd-9ddb-371c69b05e56" (UID: "00245d73-3768-4ecd-9ddb-371c69b05e56"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:13.303729 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:13.303715 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00245d73-3768-4ecd-9ddb-371c69b05e56-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00245d73-3768-4ecd-9ddb-371c69b05e56-credential-socket\") pod \"00245d73-3768-4ecd-9ddb-371c69b05e56\" (UID: \"00245d73-3768-4ecd-9ddb-371c69b05e56\") " Oct 30 21:50:13.303844 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:13.303760 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00245d73-3768-4ecd-9ddb-371c69b05e56-workload-socket" pod "00245d73-3768-4ecd-9ddb-371c69b05e56" (UID: "00245d73-3768-4ecd-9ddb-371c69b05e56"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:13.303844 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:13.303798 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00245d73-3768-4ecd-9ddb-371c69b05e56-istiod-ca-cert" pod "00245d73-3768-4ecd-9ddb-371c69b05e56" (UID: "00245d73-3768-4ecd-9ddb-371c69b05e56"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:13.303844 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:13.303802 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00245d73-3768-4ecd-9ddb-371c69b05e56-kube-api-access-kxr5d\" (UniqueName: \"kubernetes.io/projected/00245d73-3768-4ecd-9ddb-371c69b05e56-kube-api-access-kxr5d\") pod \"00245d73-3768-4ecd-9ddb-371c69b05e56\" (UID: \"00245d73-3768-4ecd-9ddb-371c69b05e56\") " Oct 30 21:50:13.304041 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:13.303862 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00245d73-3768-4ecd-9ddb-371c69b05e56-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00245d73-3768-4ecd-9ddb-371c69b05e56-istio-data\") pod \"00245d73-3768-4ecd-9ddb-371c69b05e56\" (UID: \"00245d73-3768-4ecd-9ddb-371c69b05e56\") " Oct 30 21:50:13.304098 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:13.304055 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00245d73-3768-4ecd-9ddb-371c69b05e56-credential-socket" pod "00245d73-3768-4ecd-9ddb-371c69b05e56" (UID: "00245d73-3768-4ecd-9ddb-371c69b05e56"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:13.304304 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:13.304274 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00245d73-3768-4ecd-9ddb-371c69b05e56-istio-data" pod "00245d73-3768-4ecd-9ddb-371c69b05e56" (UID: "00245d73-3768-4ecd-9ddb-371c69b05e56"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:13.304633 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:13.304608 5748 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00245d73-3768-4ecd-9ddb-371c69b05e56-credential-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:50:13.304633 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:13.304638 5748 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00245d73-3768-4ecd-9ddb-371c69b05e56-istio-data\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:50:13.304808 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:13.304652 5748 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00245d73-3768-4ecd-9ddb-371c69b05e56-workload-certs\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:50:13.304808 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:13.304666 5748 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00245d73-3768-4ecd-9ddb-371c69b05e56-istiod-ca-cert\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:50:13.304808 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:13.304680 5748 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00245d73-3768-4ecd-9ddb-371c69b05e56-workload-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:50:13.305990 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:13.305966 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00245d73-3768-4ecd-9ddb-371c69b05e56-istio-token" pod "00245d73-3768-4ecd-9ddb-371c69b05e56" (UID: "00245d73-3768-4ecd-9ddb-371c69b05e56"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:13.306088 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:13.306069 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00245d73-3768-4ecd-9ddb-371c69b05e56-kube-api-access-kxr5d" pod "00245d73-3768-4ecd-9ddb-371c69b05e56" (UID: "00245d73-3768-4ecd-9ddb-371c69b05e56"). InnerVolumeSpecName "kube-api-access-kxr5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:13.306279 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:13.306257 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00245d73-3768-4ecd-9ddb-371c69b05e56-istio-podinfo" pod "00245d73-3768-4ecd-9ddb-371c69b05e56" (UID: "00245d73-3768-4ecd-9ddb-371c69b05e56"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:50:13.306367 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:13.306254 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00245d73-3768-4ecd-9ddb-371c69b05e56-istio-envoy" pod "00245d73-3768-4ecd-9ddb-371c69b05e56" (UID: "00245d73-3768-4ecd-9ddb-371c69b05e56"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:13.407101 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:13.407065 5748 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00245d73-3768-4ecd-9ddb-371c69b05e56-istio-podinfo\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:50:13.407101 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:13.407103 5748 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00245d73-3768-4ecd-9ddb-371c69b05e56-istio-token\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:50:13.407294 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:13.407119 5748 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00245d73-3768-4ecd-9ddb-371c69b05e56-istio-envoy\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:50:13.407294 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:13.407133 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxr5d\" (UniqueName: \"kubernetes.io/projected/00245d73-3768-4ecd-9ddb-371c69b05e56-kube-api-access-kxr5d\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:50:13.802186 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:13.802152 5748 generic.go:356] "Generic (PLEG): container finished" podID="00245d73-3768-4ecd-9ddb-371c69b05e56" containerID="c37162bb04d8c3387b63a6bde42678db1be3569fb39897e7d750a707127aaa38" exitCode=0 Oct 30 21:50:13.802627 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:13.802200 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q" Oct 30 21:50:13.802627 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:13.802215 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q" event={"ID":"00245d73-3768-4ecd-9ddb-371c69b05e56","Type":"ContainerDied","Data":"c37162bb04d8c3387b63a6bde42678db1be3569fb39897e7d750a707127aaa38"} Oct 30 21:50:13.802627 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:13.802256 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q" event={"ID":"00245d73-3768-4ecd-9ddb-371c69b05e56","Type":"ContainerDied","Data":"0e4aa4645c1b2c12aad5d2ca55ca8851ee625b03b6f29c9bb03dc32509b025de"} Oct 30 21:50:13.802627 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:13.802277 5748 scope.go:122] "RemoveContainer" containerID="c37162bb04d8c3387b63a6bde42678db1be3569fb39897e7d750a707127aaa38" Oct 30 21:50:13.814512 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:13.814491 5748 scope.go:122] "RemoveContainer" containerID="c37162bb04d8c3387b63a6bde42678db1be3569fb39897e7d750a707127aaa38" Oct 30 21:50:13.814782 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:50:13.814758 5748 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c37162bb04d8c3387b63a6bde42678db1be3569fb39897e7d750a707127aaa38\": container with ID starting with c37162bb04d8c3387b63a6bde42678db1be3569fb39897e7d750a707127aaa38 not found: ID does not exist" containerID="c37162bb04d8c3387b63a6bde42678db1be3569fb39897e7d750a707127aaa38" Oct 30 21:50:13.814874 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:13.814795 5748 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c37162bb04d8c3387b63a6bde42678db1be3569fb39897e7d750a707127aaa38"} err="failed to get container status \"c37162bb04d8c3387b63a6bde42678db1be3569fb39897e7d750a707127aaa38\": rpc error: code = NotFound desc = could not find container \"c37162bb04d8c3387b63a6bde42678db1be3569fb39897e7d750a707127aaa38\": container with ID starting with c37162bb04d8c3387b63a6bde42678db1be3569fb39897e7d750a707127aaa38 not found: ID does not exist" Oct 30 21:50:13.971126 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:13.971091 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q"] Oct 30 21:50:14.015127 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:14.015085 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-xl77q"] Oct 30 21:50:14.433281 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:14.433230 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00245d73-3768-4ecd-9ddb-371c69b05e56" path="/var/lib/kubelet/pods/00245d73-3768-4ecd-9ddb-371c69b05e56/volumes" Oct 30 21:50:15.051809 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:15.051773 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60fb5a05-gdrnv"] Oct 30 21:50:15.102373 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:15.102332 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60fb5a05-gdrnv"] Oct 30 21:50:16.049316 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:16.049279 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e843d27-d4nx9"] Oct 30 21:50:16.099791 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:16.099755 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e843d27-d4nx9"] Oct 30 21:50:16.426851 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:16.426814 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59ccc507-2625-4679-b3b8-b0796c1fd01a" path="/var/lib/kubelet/pods/59ccc507-2625-4679-b3b8-b0796c1fd01a/volumes" Oct 30 21:50:16.427383 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:16.427364 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8fd0e2c-8495-4f2a-9d0e-166200ef8d59" path="/var/lib/kubelet/pods/c8fd0e2c-8495-4f2a-9d0e-166200ef8d59/volumes" Oct 30 21:50:20.063562 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:20.063520 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e86717c8-lxnvk"] Oct 30 21:50:20.111593 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:20.111560 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e86717c8-lxnvk"] Oct 30 21:50:20.418561 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:20.418525 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47875ea5-b355-4eef-af89-ff9dce4e91e7" path="/var/lib/kubelet/pods/47875ea5-b355-4eef-af89-ff9dce4e91e7/volumes" Oct 30 21:50:26.049142 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:26.049088 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a488c3f-54tr8"] Oct 30 21:50:26.099667 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:26.099635 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e7250d4-jggkz"] Oct 30 21:50:26.102971 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:26.102948 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a488c3f-54tr8"] Oct 30 21:50:26.105714 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:26.105686 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e7250d4-jggkz"] Oct 30 21:50:26.421129 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:26.421095 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="097d56ed-7515-4b94-9176-56e1c639d24c" path="/var/lib/kubelet/pods/097d56ed-7515-4b94-9176-56e1c639d24c/volumes" Oct 30 21:50:26.421624 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:26.421609 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3880a175-d299-45eb-a49a-a89b079b67e7" path="/var/lib/kubelet/pods/3880a175-d299-45eb-a49a-a89b079b67e7/volumes" Oct 30 21:50:28.048668 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:28.048635 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cae3de43-x2ltk"] Oct 30 21:50:28.097739 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:28.097704 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cae3de43-x2ltk"] Oct 30 21:50:28.418113 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:28.418078 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d77664f-6bf2-4f3a-8f07-490f1d4c12b7" path="/var/lib/kubelet/pods/0d77664f-6bf2-4f3a-8f07-490f1d4c12b7/volumes" Oct 30 21:50:30.047959 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:30.047919 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6dc779f-zzcqb"] Oct 30 21:50:30.098542 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:30.098505 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6dc779f-zzcqb"] Oct 30 21:50:30.146662 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:50:30.146626 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:50:30.418339 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:30.418308 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d335b145-924d-41c8-852f-2ebcd75c6e53" path="/var/lib/kubelet/pods/d335b145-924d-41c8-852f-2ebcd75c6e53/volumes" Oct 30 21:50:33.051538 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:33.051506 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32ad80cc-cqmzb"] Oct 30 21:50:33.055970 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:33.055945 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32ad80cc-cqmzb"] Oct 30 21:50:34.419363 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:34.419331 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99cf5ffb-5049-4f6b-aca5-4f6b22c55483" path="/var/lib/kubelet/pods/99cf5ffb-5049-4f6b-aca5-4f6b22c55483/volumes" Oct 30 21:50:36.051762 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:36.051727 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a092c8c-ftf9n"] Oct 30 21:50:36.146957 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:36.144396 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a092c8c-ftf9n"] Oct 30 21:50:36.418466 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:36.418418 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50e120fc-8f50-41eb-81f2-71e5bc8b74cf" path="/var/lib/kubelet/pods/50e120fc-8f50-41eb-81f2-71e5bc8b74cf/volumes" Oct 30 21:50:37.040340 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:37.040307 5748 scope.go:122] "RemoveContainer" containerID="86875bb2bc2a7c198275c70b35b05c42879c86f7f9ea7cf907d9f403e31b1027" Oct 30 21:50:37.050672 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:37.050649 5748 scope.go:122] "RemoveContainer" containerID="88aebcc1fb27c9a222b360aa7ef3eae00cc174ce0cdddd87da64548311e69ccf" Oct 30 21:50:37.060517 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:37.060491 5748 scope.go:122] "RemoveContainer" containerID="9ef24eb689b54cd6c4eb51440ea1594c60594f760cb6ea507385a25b6c372a36" Oct 30 21:50:37.069746 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:37.069726 5748 scope.go:122] "RemoveContainer" containerID="a506f884300f2f53a13446377bb2b173b17b41df9c559b7b4c28328cc58f11e2" Oct 30 21:50:37.081170 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:37.081141 5748 scope.go:122] "RemoveContainer" containerID="e80272bdf0b87eb5eccdb62007e6ee67761ed211bfeaddc7ae5f429a7a2cb6f6" Oct 30 21:50:37.089785 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:37.089767 5748 scope.go:122] "RemoveContainer" containerID="ff096887438d8fac25669cb0039ac32ec8bf0bb9d08c6f853bb48a322740808f" Oct 30 21:50:37.098668 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:37.098650 5748 scope.go:122] "RemoveContainer" containerID="f772945a4c29d6192d596f916ede936039cb7f4901f0a9cddc90b7b3dd75675d" Oct 30 21:50:37.107359 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:37.107337 5748 scope.go:122] "RemoveContainer" containerID="4326d9fc5b941ec3bba16215b9f0554c6687e4d9ac119659724315945a7b5632" Oct 30 21:50:39.625305 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:39.625275 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fd496e14-79mdp"] Oct 30 21:50:39.625927 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:39.625878 5748 cpu_manager.go:414] "containerMap: removing container" podUID="00245d73-3768-4ecd-9ddb-371c69b05e56" containerName="istio-proxy" Oct 30 21:50:39.625927 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:39.625907 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00245d73-3768-4ecd-9ddb-371c69b05e56" containerName="istio-proxy" Oct 30 21:50:39.626020 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:39.625932 5748 cpu_manager.go:414] "containerMap: removing container" podUID="889f25ea-9b78-4350-a0fb-770b4626a4e6" containerName="revoke-keys" Oct 30 21:50:39.626020 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:39.625938 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="889f25ea-9b78-4350-a0fb-770b4626a4e6" containerName="revoke-keys" Oct 30 21:50:39.626084 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:39.626046 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="889f25ea-9b78-4350-a0fb-770b4626a4e6" containerName="revoke-keys" Oct 30 21:50:39.626084 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:39.626063 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="00245d73-3768-4ecd-9ddb-371c69b05e56" containerName="istio-proxy" Oct 30 21:50:39.631968 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:39.631953 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fd496e14-79mdp" Oct 30 21:50:39.688091 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:39.688062 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fd496e14-79mdp"] Oct 30 21:50:39.715090 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:39.715056 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89449426-c133-4c16-89f2-20777c09b3e1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-fd496e14-79mdp\" (UID: \"89449426-c133-4c16-89f2-20777c09b3e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fd496e14-79mdp" Oct 30 21:50:39.715255 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:39.715229 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7265\" (UniqueName: \"kubernetes.io/projected/89449426-c133-4c16-89f2-20777c09b3e1-kube-api-access-w7265\") pod \"maas-api-revoke-keys-e2e-derive-fd496e14-79mdp\" (UID: \"89449426-c133-4c16-89f2-20777c09b3e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fd496e14-79mdp" Oct 30 21:50:39.715322 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:39.715287 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89449426-c133-4c16-89f2-20777c09b3e1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-fd496e14-79mdp\" (UID: \"89449426-c133-4c16-89f2-20777c09b3e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fd496e14-79mdp" Oct 30 21:50:39.734310 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:39.734283 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:50:39.734435 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:39.734362 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 21:50:39.734435 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:39.734373 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:50:39.735383 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:39.735355 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:50:39.817065 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:39.817030 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7265\" (UniqueName: \"kubernetes.io/projected/89449426-c133-4c16-89f2-20777c09b3e1-kube-api-access-w7265\") pod \"maas-api-revoke-keys-e2e-derive-fd496e14-79mdp\" (UID: \"89449426-c133-4c16-89f2-20777c09b3e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fd496e14-79mdp" Oct 30 21:50:39.817253 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:39.817105 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89449426-c133-4c16-89f2-20777c09b3e1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-fd496e14-79mdp\" (UID: \"89449426-c133-4c16-89f2-20777c09b3e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fd496e14-79mdp" Oct 30 21:50:39.817348 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:39.817264 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89449426-c133-4c16-89f2-20777c09b3e1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-fd496e14-79mdp\" (UID: \"89449426-c133-4c16-89f2-20777c09b3e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fd496e14-79mdp" Oct 30 21:50:39.817969 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:39.817935 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89449426-c133-4c16-89f2-20777c09b3e1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-fd496e14-79mdp\" (UID: \"89449426-c133-4c16-89f2-20777c09b3e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fd496e14-79mdp" Oct 30 21:50:39.819888 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:39.819848 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89449426-c133-4c16-89f2-20777c09b3e1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-fd496e14-79mdp\" (UID: \"89449426-c133-4c16-89f2-20777c09b3e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fd496e14-79mdp" Oct 30 21:50:39.878890 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:39.878815 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7265\" (UniqueName: \"kubernetes.io/projected/89449426-c133-4c16-89f2-20777c09b3e1-kube-api-access-w7265\") pod \"maas-api-revoke-keys-e2e-derive-fd496e14-79mdp\" (UID: \"89449426-c133-4c16-89f2-20777c09b3e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fd496e14-79mdp" Oct 30 21:50:39.945846 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:39.945812 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fd496e14-79mdp" Oct 30 21:50:40.357320 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:40.357285 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fd496e14-79mdp"] Oct 30 21:50:40.359094 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:50:40.359062 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89449426_c133_4c16_89f2_20777c09b3e1.slice/crio-3fcf3216c80be35632602dc329d4685eaa75acca186983153942961d827a6815 WatchSource:0}: Error finding container 3fcf3216c80be35632602dc329d4685eaa75acca186983153942961d827a6815: Status 404 returned error can't find the container with id 3fcf3216c80be35632602dc329d4685eaa75acca186983153942961d827a6815 Oct 30 21:50:41.063410 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:41.063372 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fd496e14-79mdp" event={"ID":"89449426-c133-4c16-89f2-20777c09b3e1","Type":"ContainerStarted","Data":"9fae17f77b9fdead800555a8766dbd87f37d02a5af6d679fc193ad021a68a5a3"} Oct 30 21:50:41.063813 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:41.063434 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fd496e14-79mdp" event={"ID":"89449426-c133-4c16-89f2-20777c09b3e1","Type":"ContainerStarted","Data":"3fcf3216c80be35632602dc329d4685eaa75acca186983153942961d827a6815"} Oct 30 21:50:42.074417 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:42.074315 5748 generic.go:356] "Generic (PLEG): container finished" podID="89449426-c133-4c16-89f2-20777c09b3e1" containerID="9fae17f77b9fdead800555a8766dbd87f37d02a5af6d679fc193ad021a68a5a3" exitCode=0 Oct 30 21:50:42.074417 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:42.074372 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fd496e14-79mdp" event={"ID":"89449426-c133-4c16-89f2-20777c09b3e1","Type":"ContainerDied","Data":"9fae17f77b9fdead800555a8766dbd87f37d02a5af6d679fc193ad021a68a5a3"} Oct 30 21:50:43.047650 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:43.047617 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf97b89a-qdkwh"] Oct 30 21:50:43.050808 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:43.050780 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf97b89a-qdkwh"] Oct 30 21:50:43.268409 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:43.268385 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fd496e14-79mdp" Oct 30 21:50:43.373649 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:43.373572 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89449426-c133-4c16-89f2-20777c09b3e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89449426-c133-4c16-89f2-20777c09b3e1-maas-api-openshift-ca\") pod \"89449426-c133-4c16-89f2-20777c09b3e1\" (UID: \"89449426-c133-4c16-89f2-20777c09b3e1\") " Oct 30 21:50:43.373649 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:43.373637 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89449426-c133-4c16-89f2-20777c09b3e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89449426-c133-4c16-89f2-20777c09b3e1-maas-api-xks-ca\") pod \"89449426-c133-4c16-89f2-20777c09b3e1\" (UID: \"89449426-c133-4c16-89f2-20777c09b3e1\") " Oct 30 21:50:43.373885 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:43.373753 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89449426-c133-4c16-89f2-20777c09b3e1-kube-api-access-w7265\" (UniqueName: \"kubernetes.io/projected/89449426-c133-4c16-89f2-20777c09b3e1-kube-api-access-w7265\") pod \"89449426-c133-4c16-89f2-20777c09b3e1\" (UID: \"89449426-c133-4c16-89f2-20777c09b3e1\") " Oct 30 21:50:43.374011 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:43.373982 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89449426-c133-4c16-89f2-20777c09b3e1-maas-api-openshift-ca" pod "89449426-c133-4c16-89f2-20777c09b3e1" (UID: "89449426-c133-4c16-89f2-20777c09b3e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:43.374349 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:43.374326 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89449426-c133-4c16-89f2-20777c09b3e1-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:50:43.375820 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:43.375794 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89449426-c133-4c16-89f2-20777c09b3e1-maas-api-xks-ca" pod "89449426-c133-4c16-89f2-20777c09b3e1" (UID: "89449426-c133-4c16-89f2-20777c09b3e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:50:43.375944 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:43.375833 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89449426-c133-4c16-89f2-20777c09b3e1-kube-api-access-w7265" pod "89449426-c133-4c16-89f2-20777c09b3e1" (UID: "89449426-c133-4c16-89f2-20777c09b3e1"). InnerVolumeSpecName "kube-api-access-w7265". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:43.476553 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:43.476517 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89449426-c133-4c16-89f2-20777c09b3e1-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:50:43.476553 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:43.476557 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7265\" (UniqueName: \"kubernetes.io/projected/89449426-c133-4c16-89f2-20777c09b3e1-kube-api-access-w7265\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:50:44.095074 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:44.095037 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fd496e14-79mdp" Oct 30 21:50:44.095293 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:44.095073 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fd496e14-79mdp" event={"ID":"89449426-c133-4c16-89f2-20777c09b3e1","Type":"ContainerDied","Data":"3fcf3216c80be35632602dc329d4685eaa75acca186983153942961d827a6815"} Oct 30 21:50:44.095293 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:44.095128 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3fcf3216c80be35632602dc329d4685eaa75acca186983153942961d827a6815" Oct 30 21:50:44.418375 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:44.418338 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddcc2107-c0c9-4ddd-ba34-66d000b6d204" path="/var/lib/kubelet/pods/ddcc2107-c0c9-4ddd-ba34-66d000b6d204/volumes" Oct 30 21:50:45.834301 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:45.834269 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-21084e54-99vsf"] Oct 30 21:50:45.835146 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:45.835123 5748 cpu_manager.go:414] "containerMap: removing container" podUID="89449426-c133-4c16-89f2-20777c09b3e1" containerName="revoke-keys" Oct 30 21:50:45.835146 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:45.835147 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89449426-c133-4c16-89f2-20777c09b3e1" containerName="revoke-keys" Oct 30 21:50:45.835371 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:45.835357 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="89449426-c133-4c16-89f2-20777c09b3e1" containerName="revoke-keys" Oct 30 21:50:45.839235 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:45.839214 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-21084e54-99vsf" Oct 30 21:50:45.911971 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:45.911935 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-21084e54-99vsf"] Oct 30 21:50:45.912280 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:45.912246 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1cb1437-aa71-4bab-9325-b77e15af7f4b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-21084e54-99vsf\" (UID: \"e1cb1437-aa71-4bab-9325-b77e15af7f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-21084e54-99vsf" Oct 30 21:50:45.912412 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:45.912392 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wj9t\" (UniqueName: \"kubernetes.io/projected/e1cb1437-aa71-4bab-9325-b77e15af7f4b-kube-api-access-2wj9t\") pod \"maas-api-revoke-keys-e2e-migrate-21084e54-99vsf\" (UID: \"e1cb1437-aa71-4bab-9325-b77e15af7f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-21084e54-99vsf" Oct 30 21:50:45.912521 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:45.912492 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1cb1437-aa71-4bab-9325-b77e15af7f4b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-21084e54-99vsf\" (UID: \"e1cb1437-aa71-4bab-9325-b77e15af7f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-21084e54-99vsf" Oct 30 21:50:45.958924 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:45.947247 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 21:50:45.958924 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:45.947339 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:50:45.958924 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:45.947425 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:50:45.958924 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:45.947474 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:50:46.014919 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:46.014525 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1cb1437-aa71-4bab-9325-b77e15af7f4b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-21084e54-99vsf\" (UID: \"e1cb1437-aa71-4bab-9325-b77e15af7f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-21084e54-99vsf" Oct 30 21:50:46.014919 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:46.014673 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wj9t\" (UniqueName: \"kubernetes.io/projected/e1cb1437-aa71-4bab-9325-b77e15af7f4b-kube-api-access-2wj9t\") pod \"maas-api-revoke-keys-e2e-migrate-21084e54-99vsf\" (UID: \"e1cb1437-aa71-4bab-9325-b77e15af7f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-21084e54-99vsf" Oct 30 21:50:46.014919 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:46.014773 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1cb1437-aa71-4bab-9325-b77e15af7f4b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-21084e54-99vsf\" (UID: \"e1cb1437-aa71-4bab-9325-b77e15af7f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-21084e54-99vsf" Oct 30 21:50:46.039920 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:46.015768 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1cb1437-aa71-4bab-9325-b77e15af7f4b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-21084e54-99vsf\" (UID: \"e1cb1437-aa71-4bab-9325-b77e15af7f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-21084e54-99vsf" Oct 30 21:50:46.094971 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:46.094080 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wj9t\" (UniqueName: \"kubernetes.io/projected/e1cb1437-aa71-4bab-9325-b77e15af7f4b-kube-api-access-2wj9t\") pod \"maas-api-revoke-keys-e2e-migrate-21084e54-99vsf\" (UID: \"e1cb1437-aa71-4bab-9325-b77e15af7f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-21084e54-99vsf" Oct 30 21:50:46.103921 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:46.101715 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1cb1437-aa71-4bab-9325-b77e15af7f4b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-21084e54-99vsf\" (UID: \"e1cb1437-aa71-4bab-9325-b77e15af7f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-21084e54-99vsf" Oct 30 21:50:46.154717 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:46.154683 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-21084e54-99vsf" Oct 30 21:50:46.554549 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:46.554515 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-21084e54-99vsf"] Oct 30 21:50:46.555510 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:50:46.555481 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1cb1437_aa71_4bab_9325_b77e15af7f4b.slice/crio-7c04a50c2505b6b8d21709dd0adcd8d627024f8091f6947138ac3af478451e57 WatchSource:0}: Error finding container 7c04a50c2505b6b8d21709dd0adcd8d627024f8091f6947138ac3af478451e57: Status 404 returned error can't find the container with id 7c04a50c2505b6b8d21709dd0adcd8d627024f8091f6947138ac3af478451e57 Oct 30 21:50:47.133067 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:47.133033 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-21084e54-99vsf" event={"ID":"e1cb1437-aa71-4bab-9325-b77e15af7f4b","Type":"ContainerStarted","Data":"e43a96f6d8f02150bf2f68c1a92047f197e8aadc1eb516859d1a2f446e005f6d"} Oct 30 21:50:47.133455 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:47.133117 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-21084e54-99vsf" event={"ID":"e1cb1437-aa71-4bab-9325-b77e15af7f4b","Type":"ContainerStarted","Data":"7c04a50c2505b6b8d21709dd0adcd8d627024f8091f6947138ac3af478451e57"} Oct 30 21:50:47.256692 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:47.256591 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-21084e54-99vsf" podStartSLOduration=2.256576684 podStartE2EDuration="2.256576684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:47.253472017 +0000 UTC m=+1630.995992949" watchObservedRunningTime="2026-10-30 21:50:47.256576684 +0000 UTC m=+1630.999097636" Oct 30 21:50:48.142412 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:48.142375 5748 generic.go:356] "Generic (PLEG): container finished" podID="e1cb1437-aa71-4bab-9325-b77e15af7f4b" containerID="e43a96f6d8f02150bf2f68c1a92047f197e8aadc1eb516859d1a2f446e005f6d" exitCode=0 Oct 30 21:50:48.142831 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:48.142460 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-21084e54-99vsf" event={"ID":"e1cb1437-aa71-4bab-9325-b77e15af7f4b","Type":"ContainerDied","Data":"e43a96f6d8f02150bf2f68c1a92047f197e8aadc1eb516859d1a2f446e005f6d"} Oct 30 21:50:49.341455 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:49.341428 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-21084e54-99vsf" Oct 30 21:50:49.508084 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:49.507991 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1cb1437-aa71-4bab-9325-b77e15af7f4b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1cb1437-aa71-4bab-9325-b77e15af7f4b-maas-api-xks-ca\") pod \"e1cb1437-aa71-4bab-9325-b77e15af7f4b\" (UID: \"e1cb1437-aa71-4bab-9325-b77e15af7f4b\") " Oct 30 21:50:49.508256 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:49.508138 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1cb1437-aa71-4bab-9325-b77e15af7f4b-kube-api-access-2wj9t\" (UniqueName: \"kubernetes.io/projected/e1cb1437-aa71-4bab-9325-b77e15af7f4b-kube-api-access-2wj9t\") pod \"e1cb1437-aa71-4bab-9325-b77e15af7f4b\" (UID: \"e1cb1437-aa71-4bab-9325-b77e15af7f4b\") " Oct 30 21:50:49.508431 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:49.508413 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1cb1437-aa71-4bab-9325-b77e15af7f4b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1cb1437-aa71-4bab-9325-b77e15af7f4b-maas-api-openshift-ca\") pod \"e1cb1437-aa71-4bab-9325-b77e15af7f4b\" (UID: \"e1cb1437-aa71-4bab-9325-b77e15af7f4b\") " Oct 30 21:50:49.512315 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:49.512288 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1cb1437-aa71-4bab-9325-b77e15af7f4b-maas-api-openshift-ca" pod "e1cb1437-aa71-4bab-9325-b77e15af7f4b" (UID: "e1cb1437-aa71-4bab-9325-b77e15af7f4b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:49.542214 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:49.542164 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1cb1437-aa71-4bab-9325-b77e15af7f4b-kube-api-access-2wj9t" pod "e1cb1437-aa71-4bab-9325-b77e15af7f4b" (UID: "e1cb1437-aa71-4bab-9325-b77e15af7f4b"). InnerVolumeSpecName "kube-api-access-2wj9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:49.605488 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:49.605425 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1cb1437-aa71-4bab-9325-b77e15af7f4b-maas-api-xks-ca" pod "e1cb1437-aa71-4bab-9325-b77e15af7f4b" (UID: "e1cb1437-aa71-4bab-9325-b77e15af7f4b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:50:49.611957 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:49.611016 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1cb1437-aa71-4bab-9325-b77e15af7f4b-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:50:49.611957 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:49.611055 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wj9t\" (UniqueName: \"kubernetes.io/projected/e1cb1437-aa71-4bab-9325-b77e15af7f4b-kube-api-access-2wj9t\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:50:49.611957 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:49.611074 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1cb1437-aa71-4bab-9325-b77e15af7f4b-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:50:50.163172 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:50.163141 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-21084e54-99vsf" Oct 30 21:50:50.163340 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:50.163182 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-21084e54-99vsf" event={"ID":"e1cb1437-aa71-4bab-9325-b77e15af7f4b","Type":"ContainerDied","Data":"7c04a50c2505b6b8d21709dd0adcd8d627024f8091f6947138ac3af478451e57"} Oct 30 21:50:50.163340 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:50:50.163230 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c04a50c2505b6b8d21709dd0adcd8d627024f8091f6947138ac3af478451e57" Oct 30 21:51:00.147028 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:51:00.146988 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:51:03.051394 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:03.051358 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-948a35c6-pbgz4"] Oct 30 21:51:03.101682 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:03.101649 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-948a35c6-pbgz4"] Oct 30 21:51:04.417891 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:04.417856 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58893d7b-61e4-4e3a-b7fe-1f0d743d9fa6" path="/var/lib/kubelet/pods/58893d7b-61e4-4e3a-b7fe-1f0d743d9fa6/volumes" Oct 30 21:51:23.615935 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:23.615885 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx"] Oct 30 21:51:23.616495 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:23.616093 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx" podUID="6960e4e7-1cd0-48b0-85f8-c88e46102ad4" containerName="istio-proxy" containerID="cri-o://7d2f4ae4b42ef33ebb77c2f7e1bb6c474a574bcf5935639fdebe2ad105c08deb" gracePeriod=30 Oct 30 21:51:27.164197 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:27.164160 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-21084e54-99vsf"] Oct 30 21:51:27.213913 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:27.213866 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-21084e54-99vsf"] Oct 30 21:51:28.417612 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:28.417582 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1cb1437-aa71-4bab-9325-b77e15af7f4b" path="/var/lib/kubelet/pods/e1cb1437-aa71-4bab-9325-b77e15af7f4b/volumes" Oct 30 21:51:29.045074 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:29.044968 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx" Oct 30 21:51:29.174575 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:29.174544 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-credential-socket\") pod \"6960e4e7-1cd0-48b0-85f8-c88e46102ad4\" (UID: \"6960e4e7-1cd0-48b0-85f8-c88e46102ad4\") " Oct 30 21:51:29.174766 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:29.174670 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-istio-envoy\") pod \"6960e4e7-1cd0-48b0-85f8-c88e46102ad4\" (UID: \"6960e4e7-1cd0-48b0-85f8-c88e46102ad4\") " Oct 30 21:51:29.174766 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:29.174731 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-workload-certs\") pod \"6960e4e7-1cd0-48b0-85f8-c88e46102ad4\" (UID: \"6960e4e7-1cd0-48b0-85f8-c88e46102ad4\") " Oct 30 21:51:29.174933 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:29.174822 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-istio-token\" (UniqueName: \"kubernetes.io/projected/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-istio-token\") pod \"6960e4e7-1cd0-48b0-85f8-c88e46102ad4\" (UID: \"6960e4e7-1cd0-48b0-85f8-c88e46102ad4\") " Oct 30 21:51:29.174933 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:29.174846 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-credential-socket" pod "6960e4e7-1cd0-48b0-85f8-c88e46102ad4" (UID: "6960e4e7-1cd0-48b0-85f8-c88e46102ad4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:29.174933 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:29.174884 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-kube-api-access-7cp8h\" (UniqueName: \"kubernetes.io/projected/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-kube-api-access-7cp8h\") pod \"6960e4e7-1cd0-48b0-85f8-c88e46102ad4\" (UID: \"6960e4e7-1cd0-48b0-85f8-c88e46102ad4\") " Oct 30 21:51:29.175127 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:29.174975 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-istiod-ca-cert\") pod \"6960e4e7-1cd0-48b0-85f8-c88e46102ad4\" (UID: \"6960e4e7-1cd0-48b0-85f8-c88e46102ad4\") " Oct 30 21:51:29.175127 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:29.175013 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-workload-certs" pod "6960e4e7-1cd0-48b0-85f8-c88e46102ad4" (UID: "6960e4e7-1cd0-48b0-85f8-c88e46102ad4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:29.175127 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:29.175044 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-workload-socket\") pod \"6960e4e7-1cd0-48b0-85f8-c88e46102ad4\" (UID: \"6960e4e7-1cd0-48b0-85f8-c88e46102ad4\") " Oct 30 21:51:29.175127 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:29.175105 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-istio-data\") pod \"6960e4e7-1cd0-48b0-85f8-c88e46102ad4\" (UID: \"6960e4e7-1cd0-48b0-85f8-c88e46102ad4\") " Oct 30 21:51:29.175348 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:29.175140 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-istio-podinfo\") pod \"6960e4e7-1cd0-48b0-85f8-c88e46102ad4\" (UID: \"6960e4e7-1cd0-48b0-85f8-c88e46102ad4\") " Oct 30 21:51:29.175417 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:29.175378 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-istiod-ca-cert" pod "6960e4e7-1cd0-48b0-85f8-c88e46102ad4" (UID: "6960e4e7-1cd0-48b0-85f8-c88e46102ad4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:29.175747 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:29.175624 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-istio-data" pod "6960e4e7-1cd0-48b0-85f8-c88e46102ad4" (UID: "6960e4e7-1cd0-48b0-85f8-c88e46102ad4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:29.175913 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:29.175878 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-workload-socket" pod "6960e4e7-1cd0-48b0-85f8-c88e46102ad4" (UID: "6960e4e7-1cd0-48b0-85f8-c88e46102ad4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:29.175992 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:29.175893 5748 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-istiod-ca-cert\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:29.175992 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:29.175956 5748 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-istio-data\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:29.175992 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:29.175971 5748 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-credential-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:29.175992 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:29.175986 5748 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-workload-certs\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:29.177566 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:29.177533 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-istio-envoy" pod "6960e4e7-1cd0-48b0-85f8-c88e46102ad4" (UID: "6960e4e7-1cd0-48b0-85f8-c88e46102ad4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:29.177686 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:29.177577 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-kube-api-access-7cp8h" pod "6960e4e7-1cd0-48b0-85f8-c88e46102ad4" (UID: "6960e4e7-1cd0-48b0-85f8-c88e46102ad4"). InnerVolumeSpecName "kube-api-access-7cp8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:29.177686 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:29.177629 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-istio-token" pod "6960e4e7-1cd0-48b0-85f8-c88e46102ad4" (UID: "6960e4e7-1cd0-48b0-85f8-c88e46102ad4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:29.178005 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:29.177982 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-istio-podinfo" pod "6960e4e7-1cd0-48b0-85f8-c88e46102ad4" (UID: "6960e4e7-1cd0-48b0-85f8-c88e46102ad4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:51:29.278294 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:29.278204 5748 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-istio-envoy\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:29.278294 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:29.278240 5748 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-istio-token\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:29.278294 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:29.278254 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cp8h\" (UniqueName: \"kubernetes.io/projected/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-kube-api-access-7cp8h\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:29.278294 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:29.278268 5748 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-workload-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:29.278294 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:29.278282 5748 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6960e4e7-1cd0-48b0-85f8-c88e46102ad4-istio-podinfo\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:29.514277 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:29.514244 5748 generic.go:356] "Generic (PLEG): container finished" podID="6960e4e7-1cd0-48b0-85f8-c88e46102ad4" containerID="7d2f4ae4b42ef33ebb77c2f7e1bb6c474a574bcf5935639fdebe2ad105c08deb" exitCode=0 Oct 30 21:51:29.514784 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:29.514293 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx" Oct 30 21:51:29.514784 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:29.514329 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx" event={"ID":"6960e4e7-1cd0-48b0-85f8-c88e46102ad4","Type":"ContainerDied","Data":"7d2f4ae4b42ef33ebb77c2f7e1bb6c474a574bcf5935639fdebe2ad105c08deb"} Oct 30 21:51:29.514784 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:29.514397 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx" event={"ID":"6960e4e7-1cd0-48b0-85f8-c88e46102ad4","Type":"ContainerDied","Data":"1ae8a0f8ed3db4c70def66a15196f155d32ce39a2dd133cf37605cf1464d2f5e"} Oct 30 21:51:29.514784 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:29.514431 5748 scope.go:122] "RemoveContainer" containerID="7d2f4ae4b42ef33ebb77c2f7e1bb6c474a574bcf5935639fdebe2ad105c08deb" Oct 30 21:51:29.526331 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:29.526314 5748 scope.go:122] "RemoveContainer" containerID="7d2f4ae4b42ef33ebb77c2f7e1bb6c474a574bcf5935639fdebe2ad105c08deb" Oct 30 21:51:29.526625 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:51:29.526604 5748 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d2f4ae4b42ef33ebb77c2f7e1bb6c474a574bcf5935639fdebe2ad105c08deb\": container with ID starting with 7d2f4ae4b42ef33ebb77c2f7e1bb6c474a574bcf5935639fdebe2ad105c08deb not found: ID does not exist" containerID="7d2f4ae4b42ef33ebb77c2f7e1bb6c474a574bcf5935639fdebe2ad105c08deb" Oct 30 21:51:29.526724 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:29.526629 5748 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d2f4ae4b42ef33ebb77c2f7e1bb6c474a574bcf5935639fdebe2ad105c08deb"} err="failed to get container status \"7d2f4ae4b42ef33ebb77c2f7e1bb6c474a574bcf5935639fdebe2ad105c08deb\": rpc error: code = NotFound desc = could not find container \"7d2f4ae4b42ef33ebb77c2f7e1bb6c474a574bcf5935639fdebe2ad105c08deb\": container with ID starting with 7d2f4ae4b42ef33ebb77c2f7e1bb6c474a574bcf5935639fdebe2ad105c08deb not found: ID does not exist" Oct 30 21:51:29.682661 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:29.682624 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx"] Oct 30 21:51:29.689395 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:29.689356 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-znvzx"] Oct 30 21:51:30.146151 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:51:30.146109 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:51:30.418551 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:30.418474 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6960e4e7-1cd0-48b0-85f8-c88e46102ad4" path="/var/lib/kubelet/pods/6960e4e7-1cd0-48b0-85f8-c88e46102ad4/volumes" Oct 30 21:51:34.396589 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:34.396555 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fd496e14-79mdp"] Oct 30 21:51:34.448837 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:34.448805 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fd496e14-79mdp"] Oct 30 21:51:36.421577 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:36.421540 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89449426-c133-4c16-89f2-20777c09b3e1" path="/var/lib/kubelet/pods/89449426-c133-4c16-89f2-20777c09b3e1/volumes" Oct 30 21:51:37.211733 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:37.211691 5748 scope.go:122] "RemoveContainer" containerID="d96cd213fa3aa9b038f9f709fa6509d56b70730a3ae818a68770394f13833bed" Oct 30 21:51:37.222623 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:37.222599 5748 scope.go:122] "RemoveContainer" containerID="947069212c057fa8ee43d12161b6cc54b0b9c89b062ad7fcafa6570c517d0f38" Oct 30 21:51:37.232447 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:37.232419 5748 scope.go:122] "RemoveContainer" containerID="5d97bab635960b2e303e4e9b9f1e67a318514ef26d38aab60dbc218a21d45148" Oct 30 21:51:37.242700 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:37.242675 5748 scope.go:122] "RemoveContainer" containerID="0912051ed1016479be605d73ee12dc91683398351045e88f99a83f322cb6f417" Oct 30 21:51:37.253721 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:37.253702 5748 scope.go:122] "RemoveContainer" containerID="13d95474aaedca653593a2628ab64c3890827da0cf640eb0db75c537a7177bc1" Oct 30 21:51:37.263670 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:37.263645 5748 scope.go:122] "RemoveContainer" containerID="e763bf5a728913cf6824ab8015eeced7e1ea6fd290eab77f832ff20fde392ba7" Oct 30 21:51:37.272278 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:37.272260 5748 scope.go:122] "RemoveContainer" containerID="a9c24c5cb9181a2d69b1610c858d8910c6444201b11d36d3e01c0c9f38227f1d" Oct 30 21:51:37.282494 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:37.282474 5748 scope.go:122] "RemoveContainer" containerID="e2c5d989e15e098e8f8d95f8edbf3789365b4f44695ba40efd92b4cc4cfeefd4" Oct 30 21:51:37.291069 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:37.291035 5748 scope.go:122] "RemoveContainer" containerID="13da6e3babb1c15b3b00be48dd3606c2d7619469994ba1289391a5ecff6023a0" Oct 30 21:51:37.302863 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:37.302844 5748 scope.go:122] "RemoveContainer" containerID="ad0be915e62d41afba3069e592010cdadfc6fcd178d09742b76c95b4c284d3f0" Oct 30 21:51:37.316943 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:37.316925 5748 scope.go:122] "RemoveContainer" containerID="cabc765874f70aad7084fba3b8930abd9162d35fb4cf593efca21bca0f83817f" Oct 30 21:51:37.328183 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:37.328167 5748 scope.go:122] "RemoveContainer" containerID="6a13cbee880b07da5903a07708f9af93c9748134e4e509743a46363e6c164313" Oct 30 21:51:37.338234 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:37.338220 5748 scope.go:122] "RemoveContainer" containerID="5f5a3df7f6c0ed0c562b7df312cdfb9b272856555cd48c0297c61ece42ad4d4a" Oct 30 21:51:37.346640 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:37.346623 5748 scope.go:122] "RemoveContainer" containerID="e6d38f666087febb45357f2d5db5ff2bf1a371758662834f9c82de68a5e54a69" Oct 30 21:51:37.355091 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:37.355077 5748 scope.go:122] "RemoveContainer" containerID="7c2944c7d238e05045aa754e0ad50e853a0e09555d89c599cdf05d2f8779ff7f" Oct 30 21:51:37.366037 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:37.366015 5748 scope.go:122] "RemoveContainer" containerID="a5df42c28aa41493c6daf404e0bfc49a7deea5bc943e402899aa4176f7f77cd2" Oct 30 21:51:37.374553 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:37.374528 5748 scope.go:122] "RemoveContainer" containerID="576c29020555cc99c86111fdb51059344b8520258c9e8ec35644809b768aba92" Oct 30 21:51:53.751785 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:53.751754 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr"] Oct 30 21:51:53.752214 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:53.751989 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr" podUID="38375f9a-62ae-4a49-ab8d-f3074b9ac694" containerName="istio-proxy" containerID="cri-o://0411e7eb803150c3d8274fc9fe02c1cef733252134813b5cf3f404ffc54ae3aa" gracePeriod=30 Oct 30 21:51:53.801635 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:53.801606 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9"] Oct 30 21:51:53.801794 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:53.801775 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9" podUID="0018c42f-45f5-4417-8b83-6d0cd9bd3c02" containerName="istio-proxy" containerID="cri-o://7d391d65d81a93ecd71d4e52ff50c7edb2affe5ae50dbdb5c65d46108bbbd095" gracePeriod=30 Oct 30 21:51:56.721712 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:56.721672 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd"] Oct 30 21:51:56.722171 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:56.721926 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd" podUID="890f0ec3-96a5-4ea5-8137-e7166101e49b" containerName="istio-proxy" containerID="cri-o://2c57766216776521d66068bcbee99f6c0c443ad2eec51a3b2013b91d99e201d0" gracePeriod=30 Oct 30 21:51:56.770370 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:56.770340 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss"] Oct 30 21:51:56.770556 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:56.770534 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss" podUID="b589a204-6db6-477e-98ad-362af6f83b79" containerName="istio-proxy" containerID="cri-o://377bab6ee135cc461038266457bea5702b23a0426fae0b1a5467f0a2d38a0781" gracePeriod=30 Oct 30 21:51:56.782624 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:56.782589 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9"] Oct 30 21:51:56.783858 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:56.782874 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9" podUID="ab385653-5874-4072-a986-201ae88230cd" containerName="istio-proxy" containerID="cri-o://3e74f296bfde4cad229c4a81110a392480109903ea53de450cae815458a8059c" gracePeriod=30 Oct 30 21:51:56.820589 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:56.820563 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s"] Oct 30 21:51:56.820759 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:56.820741 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s" podUID="93b2f11c-31c6-49c6-a033-c79eaaa4f7c1" containerName="istio-proxy" containerID="cri-o://b132b953533742f5afb46291f61a3a6f551a2a64903c5f5c1b50e1f9abcdbcd8" gracePeriod=30 Oct 30 21:51:58.494191 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:58.494152 5748 patch_prober.go:29] interesting pod/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:51:58.494683 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:58.494211 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd" podUID="890f0ec3-96a5-4ea5-8137-e7166101e49b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:51:58.918484 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:58.918435 5748 generic.go:356] "Generic (PLEG): container finished" podID="38375f9a-62ae-4a49-ab8d-f3074b9ac694" containerID="0411e7eb803150c3d8274fc9fe02c1cef733252134813b5cf3f404ffc54ae3aa" exitCode=0 Oct 30 21:51:58.918655 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:58.918538 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr" event={"ID":"38375f9a-62ae-4a49-ab8d-f3074b9ac694","Type":"ContainerDied","Data":"0411e7eb803150c3d8274fc9fe02c1cef733252134813b5cf3f404ffc54ae3aa"} Oct 30 21:51:59.120268 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.120240 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr" Oct 30 21:51:59.134063 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.134037 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38375f9a-62ae-4a49-ab8d-f3074b9ac694-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38375f9a-62ae-4a49-ab8d-f3074b9ac694-istio-data\") pod \"38375f9a-62ae-4a49-ab8d-f3074b9ac694\" (UID: \"38375f9a-62ae-4a49-ab8d-f3074b9ac694\") " Oct 30 21:51:59.134202 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.134099 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38375f9a-62ae-4a49-ab8d-f3074b9ac694-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38375f9a-62ae-4a49-ab8d-f3074b9ac694-workload-certs\") pod \"38375f9a-62ae-4a49-ab8d-f3074b9ac694\" (UID: \"38375f9a-62ae-4a49-ab8d-f3074b9ac694\") " Oct 30 21:51:59.134202 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.134182 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38375f9a-62ae-4a49-ab8d-f3074b9ac694-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38375f9a-62ae-4a49-ab8d-f3074b9ac694-istiod-ca-cert\") pod \"38375f9a-62ae-4a49-ab8d-f3074b9ac694\" (UID: \"38375f9a-62ae-4a49-ab8d-f3074b9ac694\") " Oct 30 21:51:59.134202 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.134199 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/38375f9a-62ae-4a49-ab8d-f3074b9ac694-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38375f9a-62ae-4a49-ab8d-f3074b9ac694-istio-podinfo\") pod \"38375f9a-62ae-4a49-ab8d-f3074b9ac694\" (UID: \"38375f9a-62ae-4a49-ab8d-f3074b9ac694\") " Oct 30 21:51:59.134382 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.134233 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38375f9a-62ae-4a49-ab8d-f3074b9ac694-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38375f9a-62ae-4a49-ab8d-f3074b9ac694-credential-socket\") pod \"38375f9a-62ae-4a49-ab8d-f3074b9ac694\" (UID: \"38375f9a-62ae-4a49-ab8d-f3074b9ac694\") " Oct 30 21:51:59.134382 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.134317 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38375f9a-62ae-4a49-ab8d-f3074b9ac694-workload-certs" pod "38375f9a-62ae-4a49-ab8d-f3074b9ac694" (UID: "38375f9a-62ae-4a49-ab8d-f3074b9ac694"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:59.134382 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.134336 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38375f9a-62ae-4a49-ab8d-f3074b9ac694-istio-data" pod "38375f9a-62ae-4a49-ab8d-f3074b9ac694" (UID: "38375f9a-62ae-4a49-ab8d-f3074b9ac694"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:59.134515 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.134388 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38375f9a-62ae-4a49-ab8d-f3074b9ac694-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38375f9a-62ae-4a49-ab8d-f3074b9ac694-istio-envoy\") pod \"38375f9a-62ae-4a49-ab8d-f3074b9ac694\" (UID: \"38375f9a-62ae-4a49-ab8d-f3074b9ac694\") " Oct 30 21:51:59.134562 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.134517 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38375f9a-62ae-4a49-ab8d-f3074b9ac694-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38375f9a-62ae-4a49-ab8d-f3074b9ac694-workload-socket\") pod \"38375f9a-62ae-4a49-ab8d-f3074b9ac694\" (UID: \"38375f9a-62ae-4a49-ab8d-f3074b9ac694\") " Oct 30 21:51:59.134562 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.134544 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38375f9a-62ae-4a49-ab8d-f3074b9ac694-credential-socket" pod "38375f9a-62ae-4a49-ab8d-f3074b9ac694" (UID: "38375f9a-62ae-4a49-ab8d-f3074b9ac694"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:59.134672 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.134624 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38375f9a-62ae-4a49-ab8d-f3074b9ac694-kube-api-access-tkt6b\" (UniqueName: \"kubernetes.io/projected/38375f9a-62ae-4a49-ab8d-f3074b9ac694-kube-api-access-tkt6b\") pod \"38375f9a-62ae-4a49-ab8d-f3074b9ac694\" (UID: \"38375f9a-62ae-4a49-ab8d-f3074b9ac694\") " Oct 30 21:51:59.134733 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.134714 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38375f9a-62ae-4a49-ab8d-f3074b9ac694-istio-token\" (UniqueName: \"kubernetes.io/projected/38375f9a-62ae-4a49-ab8d-f3074b9ac694-istio-token\") pod \"38375f9a-62ae-4a49-ab8d-f3074b9ac694\" (UID: \"38375f9a-62ae-4a49-ab8d-f3074b9ac694\") " Oct 30 21:51:59.134833 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.134804 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38375f9a-62ae-4a49-ab8d-f3074b9ac694-istiod-ca-cert" pod "38375f9a-62ae-4a49-ab8d-f3074b9ac694" (UID: "38375f9a-62ae-4a49-ab8d-f3074b9ac694"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:59.134921 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.134808 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38375f9a-62ae-4a49-ab8d-f3074b9ac694-workload-socket" pod "38375f9a-62ae-4a49-ab8d-f3074b9ac694" (UID: "38375f9a-62ae-4a49-ab8d-f3074b9ac694"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:59.135394 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.135370 5748 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38375f9a-62ae-4a49-ab8d-f3074b9ac694-istio-data\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.135551 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.135399 5748 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38375f9a-62ae-4a49-ab8d-f3074b9ac694-workload-certs\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.135551 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.135415 5748 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38375f9a-62ae-4a49-ab8d-f3074b9ac694-istiod-ca-cert\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.135551 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.135430 5748 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38375f9a-62ae-4a49-ab8d-f3074b9ac694-credential-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.135551 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.135443 5748 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38375f9a-62ae-4a49-ab8d-f3074b9ac694-workload-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.137041 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.136985 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38375f9a-62ae-4a49-ab8d-f3074b9ac694-istio-podinfo" pod "38375f9a-62ae-4a49-ab8d-f3074b9ac694" (UID: "38375f9a-62ae-4a49-ab8d-f3074b9ac694"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:51:59.137143 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.137115 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38375f9a-62ae-4a49-ab8d-f3074b9ac694-kube-api-access-tkt6b" pod "38375f9a-62ae-4a49-ab8d-f3074b9ac694" (UID: "38375f9a-62ae-4a49-ab8d-f3074b9ac694"). InnerVolumeSpecName "kube-api-access-tkt6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:59.137634 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.137607 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38375f9a-62ae-4a49-ab8d-f3074b9ac694-istio-envoy" pod "38375f9a-62ae-4a49-ab8d-f3074b9ac694" (UID: "38375f9a-62ae-4a49-ab8d-f3074b9ac694"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:59.137722 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.137606 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38375f9a-62ae-4a49-ab8d-f3074b9ac694-istio-token" pod "38375f9a-62ae-4a49-ab8d-f3074b9ac694" (UID: "38375f9a-62ae-4a49-ab8d-f3074b9ac694"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:59.159720 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.159701 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9" Oct 30 21:51:59.235875 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.235851 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-istiod-ca-cert\") pod \"0018c42f-45f5-4417-8b83-6d0cd9bd3c02\" (UID: \"0018c42f-45f5-4417-8b83-6d0cd9bd3c02\") " Oct 30 21:51:59.236028 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.235890 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-istio-token\" (UniqueName: \"kubernetes.io/projected/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-istio-token\") pod \"0018c42f-45f5-4417-8b83-6d0cd9bd3c02\" (UID: \"0018c42f-45f5-4417-8b83-6d0cd9bd3c02\") " Oct 30 21:51:59.236028 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.236005 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-credential-socket\") pod \"0018c42f-45f5-4417-8b83-6d0cd9bd3c02\" (UID: \"0018c42f-45f5-4417-8b83-6d0cd9bd3c02\") " Oct 30 21:51:59.236161 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.236099 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-istio-podinfo\") pod \"0018c42f-45f5-4417-8b83-6d0cd9bd3c02\" (UID: \"0018c42f-45f5-4417-8b83-6d0cd9bd3c02\") " Oct 30 21:51:59.236242 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.236218 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-istio-envoy\") pod \"0018c42f-45f5-4417-8b83-6d0cd9bd3c02\" (UID: \"0018c42f-45f5-4417-8b83-6d0cd9bd3c02\") " Oct 30 21:51:59.236303 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.236233 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-istiod-ca-cert" pod "0018c42f-45f5-4417-8b83-6d0cd9bd3c02" (UID: "0018c42f-45f5-4417-8b83-6d0cd9bd3c02"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:59.236382 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.236356 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-credential-socket" pod "0018c42f-45f5-4417-8b83-6d0cd9bd3c02" (UID: "0018c42f-45f5-4417-8b83-6d0cd9bd3c02"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:59.236436 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.236386 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-workload-socket\") pod \"0018c42f-45f5-4417-8b83-6d0cd9bd3c02\" (UID: \"0018c42f-45f5-4417-8b83-6d0cd9bd3c02\") " Oct 30 21:51:59.236489 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.236457 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-kube-api-access-wmjvp\" (UniqueName: \"kubernetes.io/projected/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-kube-api-access-wmjvp\") pod \"0018c42f-45f5-4417-8b83-6d0cd9bd3c02\" (UID: \"0018c42f-45f5-4417-8b83-6d0cd9bd3c02\") " Oct 30 21:51:59.236583 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.236567 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-istio-data\") pod \"0018c42f-45f5-4417-8b83-6d0cd9bd3c02\" (UID: \"0018c42f-45f5-4417-8b83-6d0cd9bd3c02\") " Oct 30 21:51:59.236723 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.236599 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-workload-certs\") pod \"0018c42f-45f5-4417-8b83-6d0cd9bd3c02\" (UID: \"0018c42f-45f5-4417-8b83-6d0cd9bd3c02\") " Oct 30 21:51:59.236723 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.236638 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-workload-socket" pod "0018c42f-45f5-4417-8b83-6d0cd9bd3c02" (UID: "0018c42f-45f5-4417-8b83-6d0cd9bd3c02"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:59.236854 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.236835 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-istio-data" pod "0018c42f-45f5-4417-8b83-6d0cd9bd3c02" (UID: "0018c42f-45f5-4417-8b83-6d0cd9bd3c02"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:59.236987 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.236961 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-workload-certs" pod "0018c42f-45f5-4417-8b83-6d0cd9bd3c02" (UID: "0018c42f-45f5-4417-8b83-6d0cd9bd3c02"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:59.237252 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.237235 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkt6b\" (UniqueName: \"kubernetes.io/projected/38375f9a-62ae-4a49-ab8d-f3074b9ac694-kube-api-access-tkt6b\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.237252 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.237253 5748 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38375f9a-62ae-4a49-ab8d-f3074b9ac694-istio-token\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.237430 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.237262 5748 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-istio-data\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.237430 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.237271 5748 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-workload-certs\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.237430 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.237280 5748 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-istiod-ca-cert\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.237430 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.237288 5748 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-credential-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.237430 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.237297 5748 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38375f9a-62ae-4a49-ab8d-f3074b9ac694-istio-podinfo\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.237430 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.237311 5748 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38375f9a-62ae-4a49-ab8d-f3074b9ac694-istio-envoy\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.237430 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.237324 5748 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-workload-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.238349 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.238324 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-istio-token" pod "0018c42f-45f5-4417-8b83-6d0cd9bd3c02" (UID: "0018c42f-45f5-4417-8b83-6d0cd9bd3c02"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:59.238561 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.238534 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-istio-podinfo" pod "0018c42f-45f5-4417-8b83-6d0cd9bd3c02" (UID: "0018c42f-45f5-4417-8b83-6d0cd9bd3c02"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:51:59.238657 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.238610 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-istio-envoy" pod "0018c42f-45f5-4417-8b83-6d0cd9bd3c02" (UID: "0018c42f-45f5-4417-8b83-6d0cd9bd3c02"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:59.238994 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.238979 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-kube-api-access-wmjvp" pod "0018c42f-45f5-4417-8b83-6d0cd9bd3c02" (UID: "0018c42f-45f5-4417-8b83-6d0cd9bd3c02"). InnerVolumeSpecName "kube-api-access-wmjvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:59.339059 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.338980 5748 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-istio-podinfo\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.339059 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.339018 5748 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-istio-envoy\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.339059 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.339033 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmjvp\" (UniqueName: \"kubernetes.io/projected/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-kube-api-access-wmjvp\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.339059 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.339048 5748 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0018c42f-45f5-4417-8b83-6d0cd9bd3c02-istio-token\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.925852 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.925817 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr" Oct 30 21:51:59.925852 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.925845 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr" event={"ID":"38375f9a-62ae-4a49-ab8d-f3074b9ac694","Type":"ContainerDied","Data":"7ec5004152e6edf30df1c3307c0aede2b3084b91bf53c632fcaf522eb1dca398"} Oct 30 21:51:59.926431 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.925945 5748 scope.go:122] "RemoveContainer" containerID="0411e7eb803150c3d8274fc9fe02c1cef733252134813b5cf3f404ffc54ae3aa" Oct 30 21:51:59.927530 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.927496 5748 generic.go:356] "Generic (PLEG): container finished" podID="0018c42f-45f5-4417-8b83-6d0cd9bd3c02" containerID="7d391d65d81a93ecd71d4e52ff50c7edb2affe5ae50dbdb5c65d46108bbbd095" exitCode=0 Oct 30 21:51:59.927649 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.927570 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9" Oct 30 21:51:59.927649 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.927578 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9" event={"ID":"0018c42f-45f5-4417-8b83-6d0cd9bd3c02","Type":"ContainerDied","Data":"7d391d65d81a93ecd71d4e52ff50c7edb2affe5ae50dbdb5c65d46108bbbd095"} Oct 30 21:51:59.927728 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.927651 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9" event={"ID":"0018c42f-45f5-4417-8b83-6d0cd9bd3c02","Type":"ContainerDied","Data":"1874e21e53fc2bfa6a88e0fcb7bef4f71c305315f4f8e51b7d85e9bb89498e1d"} Oct 30 21:51:59.937759 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.937743 5748 scope.go:122] "RemoveContainer" containerID="7d391d65d81a93ecd71d4e52ff50c7edb2affe5ae50dbdb5c65d46108bbbd095" Oct 30 21:51:59.950664 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.950643 5748 scope.go:122] "RemoveContainer" containerID="7d391d65d81a93ecd71d4e52ff50c7edb2affe5ae50dbdb5c65d46108bbbd095" Oct 30 21:51:59.951107 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:51:59.951078 5748 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d391d65d81a93ecd71d4e52ff50c7edb2affe5ae50dbdb5c65d46108bbbd095\": container with ID starting with 7d391d65d81a93ecd71d4e52ff50c7edb2affe5ae50dbdb5c65d46108bbbd095 not found: ID does not exist" containerID="7d391d65d81a93ecd71d4e52ff50c7edb2affe5ae50dbdb5c65d46108bbbd095" Oct 30 21:51:59.951205 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:51:59.951118 5748 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d391d65d81a93ecd71d4e52ff50c7edb2affe5ae50dbdb5c65d46108bbbd095"} err="failed to get container status \"7d391d65d81a93ecd71d4e52ff50c7edb2affe5ae50dbdb5c65d46108bbbd095\": rpc error: code = NotFound desc = could not find container \"7d391d65d81a93ecd71d4e52ff50c7edb2affe5ae50dbdb5c65d46108bbbd095\": container with ID starting with 7d391d65d81a93ecd71d4e52ff50c7edb2affe5ae50dbdb5c65d46108bbbd095 not found: ID does not exist" Oct 30 21:52:00.100721 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:00.100628 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr"] Oct 30 21:52:00.151039 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:00.151005 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-sjnpr"] Oct 30 21:52:00.154697 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:52:00.154668 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:52:00.215955 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:00.215922 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9"] Oct 30 21:52:00.221320 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:00.221295 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-c5wt9"] Oct 30 21:52:00.418798 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:00.418768 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0018c42f-45f5-4417-8b83-6d0cd9bd3c02" path="/var/lib/kubelet/pods/0018c42f-45f5-4417-8b83-6d0cd9bd3c02/volumes" Oct 30 21:52:00.419314 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:00.419298 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38375f9a-62ae-4a49-ab8d-f3074b9ac694" path="/var/lib/kubelet/pods/38375f9a-62ae-4a49-ab8d-f3074b9ac694/volumes" Oct 30 21:52:01.949323 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:01.949280 5748 generic.go:356] "Generic (PLEG): container finished" podID="b589a204-6db6-477e-98ad-362af6f83b79" containerID="377bab6ee135cc461038266457bea5702b23a0426fae0b1a5467f0a2d38a0781" exitCode=0 Oct 30 21:52:01.949811 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:01.949442 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss" event={"ID":"b589a204-6db6-477e-98ad-362af6f83b79","Type":"ContainerDied","Data":"377bab6ee135cc461038266457bea5702b23a0426fae0b1a5467f0a2d38a0781"} Oct 30 21:52:01.996998 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:01.994190 5748 generic.go:356] "Generic (PLEG): container finished" podID="890f0ec3-96a5-4ea5-8137-e7166101e49b" containerID="2c57766216776521d66068bcbee99f6c0c443ad2eec51a3b2013b91d99e201d0" exitCode=0 Oct 30 21:52:01.996998 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:01.994222 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd" event={"ID":"890f0ec3-96a5-4ea5-8137-e7166101e49b","Type":"ContainerDied","Data":"2c57766216776521d66068bcbee99f6c0c443ad2eec51a3b2013b91d99e201d0"} Oct 30 21:52:01.996998 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:01.996531 5748 generic.go:356] "Generic (PLEG): container finished" podID="ab385653-5874-4072-a986-201ae88230cd" containerID="3e74f296bfde4cad229c4a81110a392480109903ea53de450cae815458a8059c" exitCode=0 Oct 30 21:52:01.996998 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:01.996622 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9" event={"ID":"ab385653-5874-4072-a986-201ae88230cd","Type":"ContainerDied","Data":"3e74f296bfde4cad229c4a81110a392480109903ea53de450cae815458a8059c"} Oct 30 21:52:01.998909 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:01.998870 5748 generic.go:356] "Generic (PLEG): container finished" podID="93b2f11c-31c6-49c6-a033-c79eaaa4f7c1" containerID="b132b953533742f5afb46291f61a3a6f551a2a64903c5f5c1b50e1f9abcdbcd8" exitCode=0 Oct 30 21:52:01.999035 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:01.999017 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s" event={"ID":"93b2f11c-31c6-49c6-a033-c79eaaa4f7c1","Type":"ContainerDied","Data":"b132b953533742f5afb46291f61a3a6f551a2a64903c5f5c1b50e1f9abcdbcd8"} Oct 30 21:52:02.076333 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.076296 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd" Oct 30 21:52:02.093656 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.093623 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/890f0ec3-96a5-4ea5-8137-e7166101e49b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/890f0ec3-96a5-4ea5-8137-e7166101e49b-credential-socket\") pod \"890f0ec3-96a5-4ea5-8137-e7166101e49b\" (UID: \"890f0ec3-96a5-4ea5-8137-e7166101e49b\") " Oct 30 21:52:02.093793 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.093726 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/890f0ec3-96a5-4ea5-8137-e7166101e49b-istio-token\" (UniqueName: \"kubernetes.io/projected/890f0ec3-96a5-4ea5-8137-e7166101e49b-istio-token\") pod \"890f0ec3-96a5-4ea5-8137-e7166101e49b\" (UID: \"890f0ec3-96a5-4ea5-8137-e7166101e49b\") " Oct 30 21:52:02.093793 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.093757 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/890f0ec3-96a5-4ea5-8137-e7166101e49b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/890f0ec3-96a5-4ea5-8137-e7166101e49b-workload-certs\") pod \"890f0ec3-96a5-4ea5-8137-e7166101e49b\" (UID: \"890f0ec3-96a5-4ea5-8137-e7166101e49b\") " Oct 30 21:52:02.093929 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.093847 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/890f0ec3-96a5-4ea5-8137-e7166101e49b-kube-api-access-6vxqv\" (UniqueName: \"kubernetes.io/projected/890f0ec3-96a5-4ea5-8137-e7166101e49b-kube-api-access-6vxqv\") pod \"890f0ec3-96a5-4ea5-8137-e7166101e49b\" (UID: \"890f0ec3-96a5-4ea5-8137-e7166101e49b\") " Oct 30 21:52:02.093929 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.093909 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/890f0ec3-96a5-4ea5-8137-e7166101e49b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/890f0ec3-96a5-4ea5-8137-e7166101e49b-istiod-ca-cert\") pod \"890f0ec3-96a5-4ea5-8137-e7166101e49b\" (UID: \"890f0ec3-96a5-4ea5-8137-e7166101e49b\") " Oct 30 21:52:02.094044 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.093925 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/890f0ec3-96a5-4ea5-8137-e7166101e49b-credential-socket" pod "890f0ec3-96a5-4ea5-8137-e7166101e49b" (UID: "890f0ec3-96a5-4ea5-8137-e7166101e49b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:02.094044 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.094031 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/890f0ec3-96a5-4ea5-8137-e7166101e49b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/890f0ec3-96a5-4ea5-8137-e7166101e49b-istio-envoy\") pod \"890f0ec3-96a5-4ea5-8137-e7166101e49b\" (UID: \"890f0ec3-96a5-4ea5-8137-e7166101e49b\") " Oct 30 21:52:02.094154 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.094059 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/890f0ec3-96a5-4ea5-8137-e7166101e49b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/890f0ec3-96a5-4ea5-8137-e7166101e49b-workload-socket\") pod \"890f0ec3-96a5-4ea5-8137-e7166101e49b\" (UID: \"890f0ec3-96a5-4ea5-8137-e7166101e49b\") " Oct 30 21:52:02.094154 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.094130 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/890f0ec3-96a5-4ea5-8137-e7166101e49b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/890f0ec3-96a5-4ea5-8137-e7166101e49b-istio-data\") pod \"890f0ec3-96a5-4ea5-8137-e7166101e49b\" (UID: \"890f0ec3-96a5-4ea5-8137-e7166101e49b\") " Oct 30 21:52:02.094265 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.094194 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/890f0ec3-96a5-4ea5-8137-e7166101e49b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/890f0ec3-96a5-4ea5-8137-e7166101e49b-istio-podinfo\") pod \"890f0ec3-96a5-4ea5-8137-e7166101e49b\" (UID: \"890f0ec3-96a5-4ea5-8137-e7166101e49b\") " Oct 30 21:52:02.094508 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.094127 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/890f0ec3-96a5-4ea5-8137-e7166101e49b-workload-certs" pod "890f0ec3-96a5-4ea5-8137-e7166101e49b" (UID: "890f0ec3-96a5-4ea5-8137-e7166101e49b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:02.094739 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.094659 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/890f0ec3-96a5-4ea5-8137-e7166101e49b-workload-socket" pod "890f0ec3-96a5-4ea5-8137-e7166101e49b" (UID: "890f0ec3-96a5-4ea5-8137-e7166101e49b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:02.094739 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.094594 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/890f0ec3-96a5-4ea5-8137-e7166101e49b-istiod-ca-cert" pod "890f0ec3-96a5-4ea5-8137-e7166101e49b" (UID: "890f0ec3-96a5-4ea5-8137-e7166101e49b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:52:02.095018 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.094986 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/890f0ec3-96a5-4ea5-8137-e7166101e49b-istio-data" pod "890f0ec3-96a5-4ea5-8137-e7166101e49b" (UID: "890f0ec3-96a5-4ea5-8137-e7166101e49b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:02.095086 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.095012 5748 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/890f0ec3-96a5-4ea5-8137-e7166101e49b-credential-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:02.095086 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.095035 5748 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/890f0ec3-96a5-4ea5-8137-e7166101e49b-workload-certs\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:02.095086 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.095050 5748 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/890f0ec3-96a5-4ea5-8137-e7166101e49b-istiod-ca-cert\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:02.095086 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.095065 5748 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/890f0ec3-96a5-4ea5-8137-e7166101e49b-workload-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:02.097761 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.097738 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/890f0ec3-96a5-4ea5-8137-e7166101e49b-istio-podinfo" pod "890f0ec3-96a5-4ea5-8137-e7166101e49b" (UID: "890f0ec3-96a5-4ea5-8137-e7166101e49b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:52:02.097870 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.097828 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/890f0ec3-96a5-4ea5-8137-e7166101e49b-kube-api-access-6vxqv" pod "890f0ec3-96a5-4ea5-8137-e7166101e49b" (UID: "890f0ec3-96a5-4ea5-8137-e7166101e49b"). InnerVolumeSpecName "kube-api-access-6vxqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:02.097952 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.097879 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/890f0ec3-96a5-4ea5-8137-e7166101e49b-istio-token" pod "890f0ec3-96a5-4ea5-8137-e7166101e49b" (UID: "890f0ec3-96a5-4ea5-8137-e7166101e49b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:02.098015 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.097964 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/890f0ec3-96a5-4ea5-8137-e7166101e49b-istio-envoy" pod "890f0ec3-96a5-4ea5-8137-e7166101e49b" (UID: "890f0ec3-96a5-4ea5-8137-e7166101e49b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:02.124707 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.124625 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss" Oct 30 21:52:02.182093 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.182065 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9" Oct 30 21:52:02.186370 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.186350 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s" Oct 30 21:52:02.196108 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.196087 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-istio-data\") pod \"93b2f11c-31c6-49c6-a033-c79eaaa4f7c1\" (UID: \"93b2f11c-31c6-49c6-a033-c79eaaa4f7c1\") " Oct 30 21:52:02.196207 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.196114 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b589a204-6db6-477e-98ad-362af6f83b79-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b589a204-6db6-477e-98ad-362af6f83b79-istio-envoy\") pod \"b589a204-6db6-477e-98ad-362af6f83b79\" (UID: \"b589a204-6db6-477e-98ad-362af6f83b79\") " Oct 30 21:52:02.196207 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.196148 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b589a204-6db6-477e-98ad-362af6f83b79-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b589a204-6db6-477e-98ad-362af6f83b79-workload-certs\") pod \"b589a204-6db6-477e-98ad-362af6f83b79\" (UID: \"b589a204-6db6-477e-98ad-362af6f83b79\") " Oct 30 21:52:02.196207 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.196167 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-credential-socket\") pod \"93b2f11c-31c6-49c6-a033-c79eaaa4f7c1\" (UID: \"93b2f11c-31c6-49c6-a033-c79eaaa4f7c1\") " Oct 30 21:52:02.196207 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.196189 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-istio-envoy\") pod \"93b2f11c-31c6-49c6-a033-c79eaaa4f7c1\" (UID: \"93b2f11c-31c6-49c6-a033-c79eaaa4f7c1\") " Oct 30 21:52:02.196431 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.196216 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-workload-socket\") pod \"93b2f11c-31c6-49c6-a033-c79eaaa4f7c1\" (UID: \"93b2f11c-31c6-49c6-a033-c79eaaa4f7c1\") " Oct 30 21:52:02.196431 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.196247 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab385653-5874-4072-a986-201ae88230cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab385653-5874-4072-a986-201ae88230cd-workload-certs\") pod \"ab385653-5874-4072-a986-201ae88230cd\" (UID: \"ab385653-5874-4072-a986-201ae88230cd\") " Oct 30 21:52:02.196431 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.196268 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-istio-token\" (UniqueName: \"kubernetes.io/projected/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-istio-token\") pod \"93b2f11c-31c6-49c6-a033-c79eaaa4f7c1\" (UID: \"93b2f11c-31c6-49c6-a033-c79eaaa4f7c1\") " Oct 30 21:52:02.196431 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.196298 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b589a204-6db6-477e-98ad-362af6f83b79-kube-api-access-kttcx\" (UniqueName: \"kubernetes.io/projected/b589a204-6db6-477e-98ad-362af6f83b79-kube-api-access-kttcx\") pod \"b589a204-6db6-477e-98ad-362af6f83b79\" (UID: \"b589a204-6db6-477e-98ad-362af6f83b79\") " Oct 30 21:52:02.196431 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.196342 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab385653-5874-4072-a986-201ae88230cd-istio-token\" (UniqueName: \"kubernetes.io/projected/ab385653-5874-4072-a986-201ae88230cd-istio-token\") pod \"ab385653-5874-4072-a986-201ae88230cd\" (UID: \"ab385653-5874-4072-a986-201ae88230cd\") " Oct 30 21:52:02.196431 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.196374 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab385653-5874-4072-a986-201ae88230cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab385653-5874-4072-a986-201ae88230cd-istiod-ca-cert\") pod \"ab385653-5874-4072-a986-201ae88230cd\" (UID: \"ab385653-5874-4072-a986-201ae88230cd\") " Oct 30 21:52:02.196431 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.196397 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b589a204-6db6-477e-98ad-362af6f83b79-workload-certs" pod "b589a204-6db6-477e-98ad-362af6f83b79" (UID: "b589a204-6db6-477e-98ad-362af6f83b79"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:02.196431 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.196400 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-credential-socket" pod "93b2f11c-31c6-49c6-a033-c79eaaa4f7c1" (UID: "93b2f11c-31c6-49c6-a033-c79eaaa4f7c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:02.196431 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.196417 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-istio-data" pod "93b2f11c-31c6-49c6-a033-c79eaaa4f7c1" (UID: "93b2f11c-31c6-49c6-a033-c79eaaa4f7c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:02.196890 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.196404 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab385653-5874-4072-a986-201ae88230cd-kube-api-access-hp6x2\" (UniqueName: \"kubernetes.io/projected/ab385653-5874-4072-a986-201ae88230cd-kube-api-access-hp6x2\") pod \"ab385653-5874-4072-a986-201ae88230cd\" (UID: \"ab385653-5874-4072-a986-201ae88230cd\") " Oct 30 21:52:02.196890 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.196497 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-istio-podinfo\") pod \"93b2f11c-31c6-49c6-a033-c79eaaa4f7c1\" (UID: \"93b2f11c-31c6-49c6-a033-c79eaaa4f7c1\") " Oct 30 21:52:02.196890 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.196568 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b589a204-6db6-477e-98ad-362af6f83b79-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b589a204-6db6-477e-98ad-362af6f83b79-workload-socket\") pod \"b589a204-6db6-477e-98ad-362af6f83b79\" (UID: \"b589a204-6db6-477e-98ad-362af6f83b79\") " Oct 30 21:52:02.196890 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.196610 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab385653-5874-4072-a986-201ae88230cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab385653-5874-4072-a986-201ae88230cd-istio-data\") pod \"ab385653-5874-4072-a986-201ae88230cd\" (UID: \"ab385653-5874-4072-a986-201ae88230cd\") " Oct 30 21:52:02.196890 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.196614 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab385653-5874-4072-a986-201ae88230cd-workload-certs" pod "ab385653-5874-4072-a986-201ae88230cd" (UID: "ab385653-5874-4072-a986-201ae88230cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:02.196890 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.196656 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-workload-certs\") pod \"93b2f11c-31c6-49c6-a033-c79eaaa4f7c1\" (UID: \"93b2f11c-31c6-49c6-a033-c79eaaa4f7c1\") " Oct 30 21:52:02.196890 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.196690 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab385653-5874-4072-a986-201ae88230cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab385653-5874-4072-a986-201ae88230cd-istio-envoy\") pod \"ab385653-5874-4072-a986-201ae88230cd\" (UID: \"ab385653-5874-4072-a986-201ae88230cd\") " Oct 30 21:52:02.196890 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.196735 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b589a204-6db6-477e-98ad-362af6f83b79-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b589a204-6db6-477e-98ad-362af6f83b79-istiod-ca-cert\") pod \"b589a204-6db6-477e-98ad-362af6f83b79\" (UID: \"b589a204-6db6-477e-98ad-362af6f83b79\") " Oct 30 21:52:02.196890 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.196826 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b589a204-6db6-477e-98ad-362af6f83b79-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b589a204-6db6-477e-98ad-362af6f83b79-credential-socket\") pod \"b589a204-6db6-477e-98ad-362af6f83b79\" (UID: \"b589a204-6db6-477e-98ad-362af6f83b79\") " Oct 30 21:52:02.196890 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.196863 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b589a204-6db6-477e-98ad-362af6f83b79-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b589a204-6db6-477e-98ad-362af6f83b79-istio-data\") pod \"b589a204-6db6-477e-98ad-362af6f83b79\" (UID: \"b589a204-6db6-477e-98ad-362af6f83b79\") " Oct 30 21:52:02.197441 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.196929 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-istiod-ca-cert\") pod \"93b2f11c-31c6-49c6-a033-c79eaaa4f7c1\" (UID: \"93b2f11c-31c6-49c6-a033-c79eaaa4f7c1\") " Oct 30 21:52:02.197441 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.196969 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-workload-socket" pod "93b2f11c-31c6-49c6-a033-c79eaaa4f7c1" (UID: "93b2f11c-31c6-49c6-a033-c79eaaa4f7c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:02.197441 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.196976 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-kube-api-access-jsmdv\" (UniqueName: \"kubernetes.io/projected/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-kube-api-access-jsmdv\") pod \"93b2f11c-31c6-49c6-a033-c79eaaa4f7c1\" (UID: \"93b2f11c-31c6-49c6-a033-c79eaaa4f7c1\") " Oct 30 21:52:02.197441 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.197083 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab385653-5874-4072-a986-201ae88230cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab385653-5874-4072-a986-201ae88230cd-istio-podinfo\") pod \"ab385653-5874-4072-a986-201ae88230cd\" (UID: \"ab385653-5874-4072-a986-201ae88230cd\") " Oct 30 21:52:02.197441 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.197136 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b589a204-6db6-477e-98ad-362af6f83b79-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b589a204-6db6-477e-98ad-362af6f83b79-istio-podinfo\") pod \"b589a204-6db6-477e-98ad-362af6f83b79\" (UID: \"b589a204-6db6-477e-98ad-362af6f83b79\") " Oct 30 21:52:02.197441 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.197198 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab385653-5874-4072-a986-201ae88230cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab385653-5874-4072-a986-201ae88230cd-workload-socket\") pod \"ab385653-5874-4072-a986-201ae88230cd\" (UID: \"ab385653-5874-4072-a986-201ae88230cd\") " Oct 30 21:52:02.197441 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.197238 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab385653-5874-4072-a986-201ae88230cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab385653-5874-4072-a986-201ae88230cd-credential-socket\") pod \"ab385653-5874-4072-a986-201ae88230cd\" (UID: \"ab385653-5874-4072-a986-201ae88230cd\") " Oct 30 21:52:02.197441 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.197269 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b589a204-6db6-477e-98ad-362af6f83b79-istio-token\" (UniqueName: \"kubernetes.io/projected/b589a204-6db6-477e-98ad-362af6f83b79-istio-token\") pod \"b589a204-6db6-477e-98ad-362af6f83b79\" (UID: \"b589a204-6db6-477e-98ad-362af6f83b79\") " Oct 30 21:52:02.198190 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.198164 5748 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/890f0ec3-96a5-4ea5-8137-e7166101e49b-istio-token\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:02.198291 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.198195 5748 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-istio-data\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:02.198291 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.198211 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vxqv\" (UniqueName: \"kubernetes.io/projected/890f0ec3-96a5-4ea5-8137-e7166101e49b-kube-api-access-6vxqv\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:02.198291 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.198225 5748 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b589a204-6db6-477e-98ad-362af6f83b79-workload-certs\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:02.198291 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.198239 5748 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-credential-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:02.198291 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.198256 5748 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-workload-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:02.198291 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.198272 5748 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab385653-5874-4072-a986-201ae88230cd-workload-certs\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:02.198291 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.198286 5748 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/890f0ec3-96a5-4ea5-8137-e7166101e49b-istio-envoy\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:02.198725 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.198300 5748 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/890f0ec3-96a5-4ea5-8137-e7166101e49b-istio-data\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:02.198725 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.198313 5748 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/890f0ec3-96a5-4ea5-8137-e7166101e49b-istio-podinfo\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:02.198868 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.198850 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-istio-envoy" pod "93b2f11c-31c6-49c6-a033-c79eaaa4f7c1" (UID: "93b2f11c-31c6-49c6-a033-c79eaaa4f7c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:02.198944 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.198923 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-istio-token" pod "93b2f11c-31c6-49c6-a033-c79eaaa4f7c1" (UID: "93b2f11c-31c6-49c6-a033-c79eaaa4f7c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:02.198944 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.198924 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b589a204-6db6-477e-98ad-362af6f83b79-istio-envoy" pod "b589a204-6db6-477e-98ad-362af6f83b79" (UID: "b589a204-6db6-477e-98ad-362af6f83b79"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:02.199067 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.198993 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b589a204-6db6-477e-98ad-362af6f83b79-kube-api-access-kttcx" pod "b589a204-6db6-477e-98ad-362af6f83b79" (UID: "b589a204-6db6-477e-98ad-362af6f83b79"). InnerVolumeSpecName "kube-api-access-kttcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:02.199245 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.199220 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b589a204-6db6-477e-98ad-362af6f83b79-credential-socket" pod "b589a204-6db6-477e-98ad-362af6f83b79" (UID: "b589a204-6db6-477e-98ad-362af6f83b79"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:02.199301 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.199223 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab385653-5874-4072-a986-201ae88230cd-kube-api-access-hp6x2" pod "ab385653-5874-4072-a986-201ae88230cd" (UID: "ab385653-5874-4072-a986-201ae88230cd"). InnerVolumeSpecName "kube-api-access-hp6x2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:02.199301 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.199289 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab385653-5874-4072-a986-201ae88230cd-istiod-ca-cert" pod "ab385653-5874-4072-a986-201ae88230cd" (UID: "ab385653-5874-4072-a986-201ae88230cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:52:02.199412 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.199387 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-kube-api-access-jsmdv" pod "93b2f11c-31c6-49c6-a033-c79eaaa4f7c1" (UID: "93b2f11c-31c6-49c6-a033-c79eaaa4f7c1"). InnerVolumeSpecName "kube-api-access-jsmdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:02.199472 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.199444 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b589a204-6db6-477e-98ad-362af6f83b79-istio-data" pod "b589a204-6db6-477e-98ad-362af6f83b79" (UID: "b589a204-6db6-477e-98ad-362af6f83b79"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:02.199626 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.199599 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b589a204-6db6-477e-98ad-362af6f83b79-workload-socket" pod "b589a204-6db6-477e-98ad-362af6f83b79" (UID: "b589a204-6db6-477e-98ad-362af6f83b79"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:02.199697 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.199663 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b589a204-6db6-477e-98ad-362af6f83b79-istiod-ca-cert" pod "b589a204-6db6-477e-98ad-362af6f83b79" (UID: "b589a204-6db6-477e-98ad-362af6f83b79"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:52:02.199697 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.199670 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab385653-5874-4072-a986-201ae88230cd-workload-socket" pod "ab385653-5874-4072-a986-201ae88230cd" (UID: "ab385653-5874-4072-a986-201ae88230cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:02.199808 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.199704 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-istiod-ca-cert" pod "93b2f11c-31c6-49c6-a033-c79eaaa4f7c1" (UID: "93b2f11c-31c6-49c6-a033-c79eaaa4f7c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:52:02.199931 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.199880 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab385653-5874-4072-a986-201ae88230cd-credential-socket" pod "ab385653-5874-4072-a986-201ae88230cd" (UID: "ab385653-5874-4072-a986-201ae88230cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:02.200070 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.200048 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab385653-5874-4072-a986-201ae88230cd-istio-data" pod "ab385653-5874-4072-a986-201ae88230cd" (UID: "ab385653-5874-4072-a986-201ae88230cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:02.200159 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.200138 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-workload-certs" pod "93b2f11c-31c6-49c6-a033-c79eaaa4f7c1" (UID: "93b2f11c-31c6-49c6-a033-c79eaaa4f7c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:02.200408 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.200382 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab385653-5874-4072-a986-201ae88230cd-istio-token" pod "ab385653-5874-4072-a986-201ae88230cd" (UID: "ab385653-5874-4072-a986-201ae88230cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:02.201012 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.200991 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b589a204-6db6-477e-98ad-362af6f83b79-istio-podinfo" pod "b589a204-6db6-477e-98ad-362af6f83b79" (UID: "b589a204-6db6-477e-98ad-362af6f83b79"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:52:02.201146 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.201129 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b589a204-6db6-477e-98ad-362af6f83b79-istio-token" pod "b589a204-6db6-477e-98ad-362af6f83b79" (UID: "b589a204-6db6-477e-98ad-362af6f83b79"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:02.201624 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.201604 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab385653-5874-4072-a986-201ae88230cd-istio-podinfo" pod "ab385653-5874-4072-a986-201ae88230cd" (UID: "ab385653-5874-4072-a986-201ae88230cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:52:02.201688 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.201659 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab385653-5874-4072-a986-201ae88230cd-istio-envoy" pod "ab385653-5874-4072-a986-201ae88230cd" (UID: "ab385653-5874-4072-a986-201ae88230cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:02.201790 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.201776 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-istio-podinfo" pod "93b2f11c-31c6-49c6-a033-c79eaaa4f7c1" (UID: "93b2f11c-31c6-49c6-a033-c79eaaa4f7c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:52:02.299915 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.299805 5748 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b589a204-6db6-477e-98ad-362af6f83b79-credential-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:02.299915 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.299845 5748 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b589a204-6db6-477e-98ad-362af6f83b79-istio-data\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:02.299915 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.299861 5748 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-istiod-ca-cert\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:02.299915 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.299875 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsmdv\" (UniqueName: \"kubernetes.io/projected/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-kube-api-access-jsmdv\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:02.299915 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.299891 5748 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab385653-5874-4072-a986-201ae88230cd-istio-podinfo\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:02.300247 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.299920 5748 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b589a204-6db6-477e-98ad-362af6f83b79-istio-podinfo\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:02.300247 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.299934 5748 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab385653-5874-4072-a986-201ae88230cd-workload-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:02.300247 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.299948 5748 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab385653-5874-4072-a986-201ae88230cd-credential-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:02.300247 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.299962 5748 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b589a204-6db6-477e-98ad-362af6f83b79-istio-token\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:02.300247 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.299975 5748 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b589a204-6db6-477e-98ad-362af6f83b79-istio-envoy\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:02.300247 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.299989 5748 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-istio-envoy\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:02.300247 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.300001 5748 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-istio-token\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:02.300247 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.300015 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kttcx\" (UniqueName: \"kubernetes.io/projected/b589a204-6db6-477e-98ad-362af6f83b79-kube-api-access-kttcx\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:02.300247 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.300029 5748 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab385653-5874-4072-a986-201ae88230cd-istio-token\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:02.300247 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.300043 5748 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab385653-5874-4072-a986-201ae88230cd-istiod-ca-cert\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:02.300247 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.300057 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hp6x2\" (UniqueName: \"kubernetes.io/projected/ab385653-5874-4072-a986-201ae88230cd-kube-api-access-hp6x2\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:02.300247 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.300069 5748 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-istio-podinfo\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:02.300247 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.300083 5748 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b589a204-6db6-477e-98ad-362af6f83b79-workload-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:02.300247 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.300099 5748 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab385653-5874-4072-a986-201ae88230cd-istio-data\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:02.300247 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.300113 5748 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1-workload-certs\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:02.300247 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.300128 5748 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab385653-5874-4072-a986-201ae88230cd-istio-envoy\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:02.300247 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:02.300144 5748 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b589a204-6db6-477e-98ad-362af6f83b79-istiod-ca-cert\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:03.007258 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:03.007044 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd" event={"ID":"890f0ec3-96a5-4ea5-8137-e7166101e49b","Type":"ContainerDied","Data":"08ceb122d36cf9a30130285c301cde061f7d0aa0270882bab03f263b7236385b"} Oct 30 21:52:03.007258 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:03.007132 5748 scope.go:122] "RemoveContainer" containerID="2c57766216776521d66068bcbee99f6c0c443ad2eec51a3b2013b91d99e201d0" Oct 30 21:52:03.007258 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:03.007069 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd" Oct 30 21:52:03.008838 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:03.008543 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9" Oct 30 21:52:03.008838 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:03.008564 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9" event={"ID":"ab385653-5874-4072-a986-201ae88230cd","Type":"ContainerDied","Data":"c1b49119b77c42d604c2d78eb49d0c861276965995f9e5c8051d0efaf0f1f64f"} Oct 30 21:52:03.010248 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:03.010223 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s" event={"ID":"93b2f11c-31c6-49c6-a033-c79eaaa4f7c1","Type":"ContainerDied","Data":"b19be204e900ef747036674545d0d1258756b739a6ec3de1c00c4770e46cabff"} Oct 30 21:52:03.010382 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:03.010335 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s" Oct 30 21:52:03.011875 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:03.011851 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss" event={"ID":"b589a204-6db6-477e-98ad-362af6f83b79","Type":"ContainerDied","Data":"c7e785c86513eb692053e58fbc8658aa5d832be0b76802d1b6c90e54cf323026"} Oct 30 21:52:03.011875 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:03.011867 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss" Oct 30 21:52:03.019026 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:03.019007 5748 scope.go:122] "RemoveContainer" containerID="3e74f296bfde4cad229c4a81110a392480109903ea53de450cae815458a8059c" Oct 30 21:52:03.027950 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:03.027931 5748 scope.go:122] "RemoveContainer" containerID="b132b953533742f5afb46291f61a3a6f551a2a64903c5f5c1b50e1f9abcdbcd8" Oct 30 21:52:03.036405 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:03.036389 5748 scope.go:122] "RemoveContainer" containerID="377bab6ee135cc461038266457bea5702b23a0426fae0b1a5467f0a2d38a0781" Oct 30 21:52:03.148543 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:03.148503 5748 patch_prober.go:29] interesting pod/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:52:03.148745 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:03.148576 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9" podUID="ab385653-5874-4072-a986-201ae88230cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:52:03.176015 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:03.175984 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9"] Oct 30 21:52:03.226538 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:03.226507 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-vhjp9"] Oct 30 21:52:03.291144 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:03.291068 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s"] Oct 30 21:52:03.295855 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:03.295829 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-9t74s"] Oct 30 21:52:03.412982 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:03.408725 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss"] Oct 30 21:52:03.418655 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:03.418628 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-zzkss"] Oct 30 21:52:03.529491 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:03.529450 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd"] Oct 30 21:52:03.534133 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:03.534110 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-87njd"] Oct 30 21:52:04.417859 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:04.417826 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="890f0ec3-96a5-4ea5-8137-e7166101e49b" path="/var/lib/kubelet/pods/890f0ec3-96a5-4ea5-8137-e7166101e49b/volumes" Oct 30 21:52:04.418375 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:04.418360 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93b2f11c-31c6-49c6-a033-c79eaaa4f7c1" path="/var/lib/kubelet/pods/93b2f11c-31c6-49c6-a033-c79eaaa4f7c1/volumes" Oct 30 21:52:04.418825 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:04.418811 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab385653-5874-4072-a986-201ae88230cd" path="/var/lib/kubelet/pods/ab385653-5874-4072-a986-201ae88230cd/volumes" Oct 30 21:52:04.419659 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:04.419643 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b589a204-6db6-477e-98ad-362af6f83b79" path="/var/lib/kubelet/pods/b589a204-6db6-477e-98ad-362af6f83b79/volumes" Oct 30 21:52:30.145984 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:52:30.145942 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:52:37.487269 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:37.487234 5748 scope.go:122] "RemoveContainer" containerID="16170affbad8d56e433d4ec5e75619c900e8f104d34da3d1668ae5ec5894aad8" Oct 30 21:52:37.496944 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:37.496888 5748 scope.go:122] "RemoveContainer" containerID="1b21f4b182253975068da96238dcb8f64e9bc59bd1966122e169d671ab1397e3" Oct 30 21:52:45.046702 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:45.046658 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c2588bc-xjvfc"] Oct 30 21:52:45.096942 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:45.096891 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c2588bc-xjvfc"] Oct 30 21:52:46.417520 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:46.417487 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa3a67e9-20ac-403a-981e-f426ffd13195" path="/var/lib/kubelet/pods/aa3a67e9-20ac-403a-981e-f426ffd13195/volumes" Oct 30 21:52:57.406181 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:57.406146 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226"] Oct 30 21:52:57.406718 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:57.406703 5748 cpu_manager.go:414] "containerMap: removing container" podUID="e1cb1437-aa71-4bab-9325-b77e15af7f4b" containerName="revoke-keys" Oct 30 21:52:57.406765 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:57.406719 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1cb1437-aa71-4bab-9325-b77e15af7f4b" containerName="revoke-keys" Oct 30 21:52:57.406765 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:57.406734 5748 cpu_manager.go:414] "containerMap: removing container" podUID="890f0ec3-96a5-4ea5-8137-e7166101e49b" containerName="istio-proxy" Oct 30 21:52:57.406765 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:57.406739 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="890f0ec3-96a5-4ea5-8137-e7166101e49b" containerName="istio-proxy" Oct 30 21:52:57.406765 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:57.406757 5748 cpu_manager.go:414] "containerMap: removing container" podUID="ab385653-5874-4072-a986-201ae88230cd" containerName="istio-proxy" Oct 30 21:52:57.406765 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:57.406762 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab385653-5874-4072-a986-201ae88230cd" containerName="istio-proxy" Oct 30 21:52:57.406961 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:57.406773 5748 cpu_manager.go:414] "containerMap: removing container" podUID="0018c42f-45f5-4417-8b83-6d0cd9bd3c02" containerName="istio-proxy" Oct 30 21:52:57.406961 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:57.406778 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0018c42f-45f5-4417-8b83-6d0cd9bd3c02" containerName="istio-proxy" Oct 30 21:52:57.406961 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:57.406793 5748 cpu_manager.go:414] "containerMap: removing container" podUID="b589a204-6db6-477e-98ad-362af6f83b79" containerName="istio-proxy" Oct 30 21:52:57.406961 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:57.406798 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b589a204-6db6-477e-98ad-362af6f83b79" containerName="istio-proxy" Oct 30 21:52:57.406961 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:57.406809 5748 cpu_manager.go:414] "containerMap: removing container" podUID="38375f9a-62ae-4a49-ab8d-f3074b9ac694" containerName="istio-proxy" Oct 30 21:52:57.406961 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:57.406813 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38375f9a-62ae-4a49-ab8d-f3074b9ac694" containerName="istio-proxy" Oct 30 21:52:57.406961 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:57.406819 5748 cpu_manager.go:414] "containerMap: removing container" podUID="93b2f11c-31c6-49c6-a033-c79eaaa4f7c1" containerName="istio-proxy" Oct 30 21:52:57.406961 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:57.406823 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93b2f11c-31c6-49c6-a033-c79eaaa4f7c1" containerName="istio-proxy" Oct 30 21:52:57.406961 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:57.406839 5748 cpu_manager.go:414] "containerMap: removing container" podUID="6960e4e7-1cd0-48b0-85f8-c88e46102ad4" containerName="istio-proxy" Oct 30 21:52:57.406961 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:57.406843 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6960e4e7-1cd0-48b0-85f8-c88e46102ad4" containerName="istio-proxy" Oct 30 21:52:57.406961 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:57.406936 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="6960e4e7-1cd0-48b0-85f8-c88e46102ad4" containerName="istio-proxy" Oct 30 21:52:57.406961 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:57.406945 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1cb1437-aa71-4bab-9325-b77e15af7f4b" containerName="revoke-keys" Oct 30 21:52:57.406961 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:57.406953 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="890f0ec3-96a5-4ea5-8137-e7166101e49b" containerName="istio-proxy" Oct 30 21:52:57.406961 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:57.406961 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab385653-5874-4072-a986-201ae88230cd" containerName="istio-proxy" Oct 30 21:52:57.407373 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:57.406969 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="0018c42f-45f5-4417-8b83-6d0cd9bd3c02" containerName="istio-proxy" Oct 30 21:52:57.407373 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:57.406976 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="b589a204-6db6-477e-98ad-362af6f83b79" containerName="istio-proxy" Oct 30 21:52:57.407373 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:57.406984 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="38375f9a-62ae-4a49-ab8d-f3074b9ac694" containerName="istio-proxy" Oct 30 21:52:57.407373 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:57.406990 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="93b2f11c-31c6-49c6-a033-c79eaaa4f7c1" containerName="istio-proxy" Oct 30 21:52:57.410370 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:57.410349 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226" Oct 30 21:52:57.504157 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:57.483881 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9aab906-835e-4acf-b20f-ef27d8bd8568-workload-certs\") pod \"e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226\" (UID: \"e9aab906-835e-4acf-b20f-ef27d8bd8568\") " pod="openshift-ingress/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226" Oct 30 21:52:57.579919 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:57.578114 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9aab906-835e-4acf-b20f-ef27d8bd8568-workload-socket\") pod \"e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226\" (UID: \"e9aab906-835e-4acf-b20f-ef27d8bd8568\") " pod="openshift-ingress/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226" Oct 30 21:52:57.579919 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:57.578241 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9aab906-835e-4acf-b20f-ef27d8bd8568-credential-socket\") pod \"e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226\" (UID: \"e9aab906-835e-4acf-b20f-ef27d8bd8568\") " pod="openshift-ingress/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226" Oct 30 21:52:57.579919 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:57.578276 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9aab906-835e-4acf-b20f-ef27d8bd8568-istio-data\") pod \"e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226\" (UID: \"e9aab906-835e-4acf-b20f-ef27d8bd8568\") " pod="openshift-ingress/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226" Oct 30 21:52:57.579919 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:57.578310 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9aab906-835e-4acf-b20f-ef27d8bd8568-istio-token\") pod \"e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226\" (UID: \"e9aab906-835e-4acf-b20f-ef27d8bd8568\") " pod="openshift-ingress/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226" Oct 30 21:52:57.579919 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:57.578455 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkplt\" (UniqueName: \"kubernetes.io/projected/e9aab906-835e-4acf-b20f-ef27d8bd8568-kube-api-access-nkplt\") pod \"e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226\" (UID: \"e9aab906-835e-4acf-b20f-ef27d8bd8568\") " pod="openshift-ingress/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226" Oct 30 21:52:57.579919 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:57.578501 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9aab906-835e-4acf-b20f-ef27d8bd8568-istiod-ca-cert\") pod \"e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226\" (UID: \"e9aab906-835e-4acf-b20f-ef27d8bd8568\") " pod="openshift-ingress/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226" Oct 30 21:52:57.579919 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:57.578649 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9aab906-835e-4acf-b20f-ef27d8bd8568-istio-envoy\") pod \"e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226\" (UID: \"e9aab906-835e-4acf-b20f-ef27d8bd8568\") " pod="openshift-ingress/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226" Oct 30 21:52:57.579919 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:57.578705 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9aab906-835e-4acf-b20f-ef27d8bd8568-istio-podinfo\") pod \"e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226\" (UID: \"e9aab906-835e-4acf-b20f-ef27d8bd8568\") " pod="openshift-ingress/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226" Oct 30 21:52:57.581938 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:57.581240 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-79e10a-openshift-default-dockercfg-tgwd8\"" Oct 30 21:52:57.586804 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:57.586772 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226"] Oct 30 21:52:57.679800 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:57.679716 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9aab906-835e-4acf-b20f-ef27d8bd8568-workload-certs\") pod \"e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226\" (UID: \"e9aab906-835e-4acf-b20f-ef27d8bd8568\") " pod="openshift-ingress/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226" Oct 30 21:52:57.679976 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:57.679869 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9aab906-835e-4acf-b20f-ef27d8bd8568-workload-socket\") pod \"e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226\" (UID: \"e9aab906-835e-4acf-b20f-ef27d8bd8568\") " pod="openshift-ingress/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226" Oct 30 21:52:57.680042 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:57.679972 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9aab906-835e-4acf-b20f-ef27d8bd8568-credential-socket\") pod \"e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226\" (UID: \"e9aab906-835e-4acf-b20f-ef27d8bd8568\") " pod="openshift-ingress/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226" Oct 30 21:52:57.680042 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:57.680001 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9aab906-835e-4acf-b20f-ef27d8bd8568-istio-data\") pod \"e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226\" (UID: \"e9aab906-835e-4acf-b20f-ef27d8bd8568\") " pod="openshift-ingress/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226" Oct 30 21:52:57.680042 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:57.680033 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9aab906-835e-4acf-b20f-ef27d8bd8568-istio-token\") pod \"e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226\" (UID: \"e9aab906-835e-4acf-b20f-ef27d8bd8568\") " pod="openshift-ingress/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226" Oct 30 21:52:57.680215 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:57.680116 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkplt\" (UniqueName: \"kubernetes.io/projected/e9aab906-835e-4acf-b20f-ef27d8bd8568-kube-api-access-nkplt\") pod \"e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226\" (UID: \"e9aab906-835e-4acf-b20f-ef27d8bd8568\") " pod="openshift-ingress/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226" Oct 30 21:52:57.680215 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:57.680136 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9aab906-835e-4acf-b20f-ef27d8bd8568-workload-certs\") pod \"e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226\" (UID: \"e9aab906-835e-4acf-b20f-ef27d8bd8568\") " pod="openshift-ingress/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226" Oct 30 21:52:57.680215 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:57.680176 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9aab906-835e-4acf-b20f-ef27d8bd8568-istiod-ca-cert\") pod \"e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226\" (UID: \"e9aab906-835e-4acf-b20f-ef27d8bd8568\") " pod="openshift-ingress/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226" Oct 30 21:52:57.680386 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:57.680218 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9aab906-835e-4acf-b20f-ef27d8bd8568-workload-socket\") pod \"e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226\" (UID: \"e9aab906-835e-4acf-b20f-ef27d8bd8568\") " pod="openshift-ingress/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226" Oct 30 21:52:57.680386 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:57.680285 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9aab906-835e-4acf-b20f-ef27d8bd8568-istio-data\") pod \"e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226\" (UID: \"e9aab906-835e-4acf-b20f-ef27d8bd8568\") " pod="openshift-ingress/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226" Oct 30 21:52:57.680386 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:57.680313 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9aab906-835e-4acf-b20f-ef27d8bd8568-istio-envoy\") pod \"e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226\" (UID: \"e9aab906-835e-4acf-b20f-ef27d8bd8568\") " pod="openshift-ingress/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226" Oct 30 21:52:57.680386 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:57.680360 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9aab906-835e-4acf-b20f-ef27d8bd8568-istio-podinfo\") pod \"e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226\" (UID: \"e9aab906-835e-4acf-b20f-ef27d8bd8568\") " pod="openshift-ingress/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226" Oct 30 21:52:57.680629 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:57.680357 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9aab906-835e-4acf-b20f-ef27d8bd8568-credential-socket\") pod \"e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226\" (UID: \"e9aab906-835e-4acf-b20f-ef27d8bd8568\") " pod="openshift-ingress/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226" Oct 30 21:52:57.680975 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:57.680952 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9aab906-835e-4acf-b20f-ef27d8bd8568-istiod-ca-cert\") pod \"e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226\" (UID: \"e9aab906-835e-4acf-b20f-ef27d8bd8568\") " pod="openshift-ingress/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226" Oct 30 21:52:57.682676 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:57.682656 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9aab906-835e-4acf-b20f-ef27d8bd8568-istio-podinfo\") pod \"e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226\" (UID: \"e9aab906-835e-4acf-b20f-ef27d8bd8568\") " pod="openshift-ingress/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226" Oct 30 21:52:57.682768 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:57.682695 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9aab906-835e-4acf-b20f-ef27d8bd8568-istio-envoy\") pod \"e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226\" (UID: \"e9aab906-835e-4acf-b20f-ef27d8bd8568\") " pod="openshift-ingress/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226" Oct 30 21:52:57.742658 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:57.742614 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkplt\" (UniqueName: \"kubernetes.io/projected/e9aab906-835e-4acf-b20f-ef27d8bd8568-kube-api-access-nkplt\") pod \"e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226\" (UID: \"e9aab906-835e-4acf-b20f-ef27d8bd8568\") " pod="openshift-ingress/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226" Oct 30 21:52:57.747374 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:57.747341 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9aab906-835e-4acf-b20f-ef27d8bd8568-istio-token\") pod \"e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226\" (UID: \"e9aab906-835e-4acf-b20f-ef27d8bd8568\") " pod="openshift-ingress/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226" Oct 30 21:52:58.023722 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:58.023631 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226" Oct 30 21:52:58.463380 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:52:58.463340 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9aab906_835e_4acf_b20f_ef27d8bd8568.slice/crio-90760c108a0714d9fee2863f1d69ca1edae0da79872cd730c8a3b5b712e3a739 WatchSource:0}: Error finding container 90760c108a0714d9fee2863f1d69ca1edae0da79872cd730c8a3b5b712e3a739: Status 404 returned error can't find the container with id 90760c108a0714d9fee2863f1d69ca1edae0da79872cd730c8a3b5b712e3a739 Oct 30 21:52:58.463691 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:58.463401 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226"] Oct 30 21:52:58.466075 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:58.466044 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:52:58.466159 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:58.466109 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:52:58.466214 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:58.466158 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:52:59.415425 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:59.415389 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226" event={"ID":"e9aab906-835e-4acf-b20f-ef27d8bd8568","Type":"ContainerStarted","Data":"36b317e896eca6413b3fcf645a68946f9e9ae4a601b45ca6f7be2a2c3504adfe"} Oct 30 21:52:59.415641 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:59.415454 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226" event={"ID":"e9aab906-835e-4acf-b20f-ef27d8bd8568","Type":"ContainerStarted","Data":"90760c108a0714d9fee2863f1d69ca1edae0da79872cd730c8a3b5b712e3a739"} Oct 30 21:52:59.591408 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:52:59.591347 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226" podStartSLOduration=2.591327015 podStartE2EDuration="2.591327015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:59.587584863 +0000 UTC m=+1763.330105811" watchObservedRunningTime="2026-10-30 21:52:59.591327015 +0000 UTC m=+1763.333847950" Oct 30 21:53:00.024810 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:53:00.024775 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226" Oct 30 21:53:00.030277 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:53:00.030248 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226" Oct 30 21:53:00.144975 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:53:00.144943 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:53:00.422600 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:53:00.422570 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226" Oct 30 21:53:00.423598 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:53:00.423573 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226" Oct 30 21:53:06.303037 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:53:06.303004 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-79e10a-55575bc788-b6qw2"] Oct 30 21:53:06.307407 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:53:06.307390 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-79e10a-55575bc788-b6qw2" Oct 30 21:53:06.398084 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:53:06.391641 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-79e10a-55575bc788-b6qw2"] Oct 30 21:53:06.447985 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:53:06.446746 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-79e10a-dockercfg-wdtlh\"" Oct 30 21:53:06.449255 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:53:06.449229 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-79e10a\"" Oct 30 21:53:06.487095 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:53:06.487051 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qkl8\" (UniqueName: \"kubernetes.io/projected/621fdae0-e3ad-43a6-a556-b7c55979f081-kube-api-access-8qkl8\") pod \"payload-processing-e2e-api-a-79e10a-55575bc788-b6qw2\" (UID: \"621fdae0-e3ad-43a6-a556-b7c55979f081\") " pod="openshift-ingress/payload-processing-e2e-api-a-79e10a-55575bc788-b6qw2" Oct 30 21:53:06.487433 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:53:06.487409 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/621fdae0-e3ad-43a6-a556-b7c55979f081-plugins-config-volume\") pod \"payload-processing-e2e-api-a-79e10a-55575bc788-b6qw2\" (UID: \"621fdae0-e3ad-43a6-a556-b7c55979f081\") " pod="openshift-ingress/payload-processing-e2e-api-a-79e10a-55575bc788-b6qw2" Oct 30 21:53:06.589298 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:53:06.589193 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/621fdae0-e3ad-43a6-a556-b7c55979f081-plugins-config-volume\") pod \"payload-processing-e2e-api-a-79e10a-55575bc788-b6qw2\" (UID: \"621fdae0-e3ad-43a6-a556-b7c55979f081\") " pod="openshift-ingress/payload-processing-e2e-api-a-79e10a-55575bc788-b6qw2" Oct 30 21:53:06.589298 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:53:06.589250 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qkl8\" (UniqueName: \"kubernetes.io/projected/621fdae0-e3ad-43a6-a556-b7c55979f081-kube-api-access-8qkl8\") pod \"payload-processing-e2e-api-a-79e10a-55575bc788-b6qw2\" (UID: \"621fdae0-e3ad-43a6-a556-b7c55979f081\") " pod="openshift-ingress/payload-processing-e2e-api-a-79e10a-55575bc788-b6qw2" Oct 30 21:53:06.589949 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:53:06.589927 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/621fdae0-e3ad-43a6-a556-b7c55979f081-plugins-config-volume\") pod \"payload-processing-e2e-api-a-79e10a-55575bc788-b6qw2\" (UID: \"621fdae0-e3ad-43a6-a556-b7c55979f081\") " pod="openshift-ingress/payload-processing-e2e-api-a-79e10a-55575bc788-b6qw2" Oct 30 21:53:06.651563 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:53:06.651531 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qkl8\" (UniqueName: \"kubernetes.io/projected/621fdae0-e3ad-43a6-a556-b7c55979f081-kube-api-access-8qkl8\") pod \"payload-processing-e2e-api-a-79e10a-55575bc788-b6qw2\" (UID: \"621fdae0-e3ad-43a6-a556-b7c55979f081\") " pod="openshift-ingress/payload-processing-e2e-api-a-79e10a-55575bc788-b6qw2" Oct 30 21:53:06.922016 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:53:06.921984 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-79e10a-55575bc788-b6qw2" Oct 30 21:53:07.307339 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:53:07.306409 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-79e10a-55575bc788-b6qw2"] Oct 30 21:53:07.514000 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:53:07.513969 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-79e10a-55575bc788-b6qw2" event={"ID":"621fdae0-e3ad-43a6-a556-b7c55979f081","Type":"ContainerStarted","Data":"5540fe39816fa508dbc08d8f263ebaf35d5770b25caca2eb327475d6d860aefa"} Oct 30 21:53:07.514148 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:53:07.514031 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-79e10a-55575bc788-b6qw2" event={"ID":"621fdae0-e3ad-43a6-a556-b7c55979f081","Type":"ContainerStarted","Data":"e9fcfb91b606bbd8d8f75e0d9e603236396dad7336ec56124ee43bfc467082be"} Oct 30 21:53:07.514202 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:53:07.514160 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-79e10a-55575bc788-b6qw2" Oct 30 21:53:07.634054 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:53:07.633963 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-79e10a-55575bc788-b6qw2" podStartSLOduration=1.633951171 podStartE2EDuration="1.633951171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:07.631351442 +0000 UTC m=+1771.373872376" watchObservedRunningTime="2026-10-30 21:53:07.633951171 +0000 UTC m=+1771.376472105" Oct 30 21:53:18.524524 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:53:18.524488 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-79e10a-55575bc788-b6qw2" Oct 30 21:53:30.144984 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:53:30.144944 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:53:36.729125 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:53:36.729092 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j4v8r_3df036ff-1c00-4ce2-9e86-8acbeedf6c82/ovn-acl-logging/0.log" Oct 30 21:53:36.731028 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:53:36.731008 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j4v8r_3df036ff-1c00-4ce2-9e86-8acbeedf6c82/ovn-acl-logging/0.log" Oct 30 21:53:42.897026 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:53:42.885355 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-bd3c59-55b47bcd98-qtjwp"] Oct 30 21:53:42.905604 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:53:42.905576 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-bd3c59-55b47bcd98-qtjwp"] Oct 30 21:53:42.905745 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:53:42.905643 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-bd3c59-55b47bcd98-qtjwp" Oct 30 21:53:43.000270 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:53:43.000235 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2lsv\" (UniqueName: \"kubernetes.io/projected/fae90fa9-3d3c-4b91-ba40-b19bf6340d17-kube-api-access-m2lsv\") pod \"payload-processing-e2e-shared-b-w2-bd3c59-55b47bcd98-qtjwp\" (UID: \"fae90fa9-3d3c-4b91-ba40-b19bf6340d17\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-bd3c59-55b47bcd98-qtjwp" Oct 30 21:53:43.000475 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:53:43.000408 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fae90fa9-3d3c-4b91-ba40-b19bf6340d17-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-bd3c59-55b47bcd98-qtjwp\" (UID: \"fae90fa9-3d3c-4b91-ba40-b19bf6340d17\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-bd3c59-55b47bcd98-qtjwp" Oct 30 21:53:43.011293 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:53:43.011268 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-bd3c59\"" Oct 30 21:53:43.011293 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:53:43.011268 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-bd3c59-dockercfg-nl65v\"" Oct 30 21:53:43.101628 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:53:43.101596 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fae90fa9-3d3c-4b91-ba40-b19bf6340d17-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-bd3c59-55b47bcd98-qtjwp\" (UID: \"fae90fa9-3d3c-4b91-ba40-b19bf6340d17\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-bd3c59-55b47bcd98-qtjwp" Oct 30 21:53:43.101784 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:53:43.101741 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2lsv\" (UniqueName: \"kubernetes.io/projected/fae90fa9-3d3c-4b91-ba40-b19bf6340d17-kube-api-access-m2lsv\") pod \"payload-processing-e2e-shared-b-w2-bd3c59-55b47bcd98-qtjwp\" (UID: \"fae90fa9-3d3c-4b91-ba40-b19bf6340d17\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-bd3c59-55b47bcd98-qtjwp" Oct 30 21:53:43.102322 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:53:43.102303 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fae90fa9-3d3c-4b91-ba40-b19bf6340d17-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-bd3c59-55b47bcd98-qtjwp\" (UID: \"fae90fa9-3d3c-4b91-ba40-b19bf6340d17\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-bd3c59-55b47bcd98-qtjwp" Oct 30 21:53:43.167496 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:53:43.167418 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2lsv\" (UniqueName: \"kubernetes.io/projected/fae90fa9-3d3c-4b91-ba40-b19bf6340d17-kube-api-access-m2lsv\") pod \"payload-processing-e2e-shared-b-w2-bd3c59-55b47bcd98-qtjwp\" (UID: \"fae90fa9-3d3c-4b91-ba40-b19bf6340d17\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-bd3c59-55b47bcd98-qtjwp" Oct 30 21:53:43.240563 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:53:43.240532 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-bd3c59-55b47bcd98-qtjwp" Oct 30 21:53:43.621079 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:53:43.621048 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-bd3c59-55b47bcd98-qtjwp"] Oct 30 21:53:43.622196 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:53:43.622168 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfae90fa9_3d3c_4b91_ba40_b19bf6340d17.slice/crio-016701d0783cac5947586eebbbe6527d8fa9a857b925a1a3383c7c1d15727ff4 WatchSource:0}: Error finding container 016701d0783cac5947586eebbbe6527d8fa9a857b925a1a3383c7c1d15727ff4: Status 404 returned error can't find the container with id 016701d0783cac5947586eebbbe6527d8fa9a857b925a1a3383c7c1d15727ff4 Oct 30 21:53:43.980808 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:53:43.980767 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-bd3c59-55b47bcd98-qtjwp" event={"ID":"fae90fa9-3d3c-4b91-ba40-b19bf6340d17","Type":"ContainerStarted","Data":"9a68e2981606c5cae4e8571b3b577b87a73c1df1ad03cb75822974093c574138"} Oct 30 21:53:43.981306 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:53:43.980835 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-bd3c59-55b47bcd98-qtjwp" event={"ID":"fae90fa9-3d3c-4b91-ba40-b19bf6340d17","Type":"ContainerStarted","Data":"016701d0783cac5947586eebbbe6527d8fa9a857b925a1a3383c7c1d15727ff4"} Oct 30 21:53:43.981306 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:53:43.980856 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-bd3c59-55b47bcd98-qtjwp" Oct 30 21:53:44.102924 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:53:44.102858 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-bd3c59-55b47bcd98-qtjwp" podStartSLOduration=2.102845164 podStartE2EDuration="2.102845164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:44.100483209 +0000 UTC m=+1807.843004158" watchObservedRunningTime="2026-10-30 21:53:44.102845164 +0000 UTC m=+1807.845366117" Oct 30 21:53:54.710860 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:53:54.710780 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m"] Oct 30 21:53:54.711353 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:53:54.711034 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m" podUID="39fe3952-ddaf-4858-af83-458d1c01f0e8" containerName="istio-proxy" containerID="cri-o://b3989391e8071317375a147526b7af75d2101b4587f0de7ec89098f2d587b0bf" gracePeriod=30 Oct 30 21:53:54.990704 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:53:54.990614 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-bd3c59-55b47bcd98-qtjwp" Oct 30 21:53:59.391532 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:53:59.391473 5748 patch_prober.go:29] interesting pod/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:53:59.391958 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:53:59.391570 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m" podUID="39fe3952-ddaf-4858-af83-458d1c01f0e8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:54:00.004767 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:00.004378 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m" Oct 30 21:54:00.069255 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:00.069217 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/39fe3952-ddaf-4858-af83-458d1c01f0e8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39fe3952-ddaf-4858-af83-458d1c01f0e8-istio-podinfo\") pod \"39fe3952-ddaf-4858-af83-458d1c01f0e8\" (UID: \"39fe3952-ddaf-4858-af83-458d1c01f0e8\") " Oct 30 21:54:00.069451 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:00.069312 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39fe3952-ddaf-4858-af83-458d1c01f0e8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39fe3952-ddaf-4858-af83-458d1c01f0e8-credential-socket\") pod \"39fe3952-ddaf-4858-af83-458d1c01f0e8\" (UID: \"39fe3952-ddaf-4858-af83-458d1c01f0e8\") " Oct 30 21:54:00.069451 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:00.069358 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39fe3952-ddaf-4858-af83-458d1c01f0e8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39fe3952-ddaf-4858-af83-458d1c01f0e8-istiod-ca-cert\") pod \"39fe3952-ddaf-4858-af83-458d1c01f0e8\" (UID: \"39fe3952-ddaf-4858-af83-458d1c01f0e8\") " Oct 30 21:54:00.069580 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:00.069462 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39fe3952-ddaf-4858-af83-458d1c01f0e8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39fe3952-ddaf-4858-af83-458d1c01f0e8-workload-certs\") pod \"39fe3952-ddaf-4858-af83-458d1c01f0e8\" (UID: \"39fe3952-ddaf-4858-af83-458d1c01f0e8\") " Oct 30 21:54:00.069580 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:00.069520 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39fe3952-ddaf-4858-af83-458d1c01f0e8-istio-token\" (UniqueName: \"kubernetes.io/projected/39fe3952-ddaf-4858-af83-458d1c01f0e8-istio-token\") pod \"39fe3952-ddaf-4858-af83-458d1c01f0e8\" (UID: \"39fe3952-ddaf-4858-af83-458d1c01f0e8\") " Oct 30 21:54:00.069696 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:00.069638 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39fe3952-ddaf-4858-af83-458d1c01f0e8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39fe3952-ddaf-4858-af83-458d1c01f0e8-istio-data\") pod \"39fe3952-ddaf-4858-af83-458d1c01f0e8\" (UID: \"39fe3952-ddaf-4858-af83-458d1c01f0e8\") " Oct 30 21:54:00.069752 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:00.069664 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39fe3952-ddaf-4858-af83-458d1c01f0e8-credential-socket" pod "39fe3952-ddaf-4858-af83-458d1c01f0e8" (UID: "39fe3952-ddaf-4858-af83-458d1c01f0e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:00.069752 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:00.069722 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39fe3952-ddaf-4858-af83-458d1c01f0e8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39fe3952-ddaf-4858-af83-458d1c01f0e8-workload-socket\") pod \"39fe3952-ddaf-4858-af83-458d1c01f0e8\" (UID: \"39fe3952-ddaf-4858-af83-458d1c01f0e8\") " Oct 30 21:54:00.069752 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:00.069744 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39fe3952-ddaf-4858-af83-458d1c01f0e8-kube-api-access-tqc2m\" (UniqueName: \"kubernetes.io/projected/39fe3952-ddaf-4858-af83-458d1c01f0e8-kube-api-access-tqc2m\") pod \"39fe3952-ddaf-4858-af83-458d1c01f0e8\" (UID: \"39fe3952-ddaf-4858-af83-458d1c01f0e8\") " Oct 30 21:54:00.069875 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:00.069741 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39fe3952-ddaf-4858-af83-458d1c01f0e8-workload-certs" pod "39fe3952-ddaf-4858-af83-458d1c01f0e8" (UID: "39fe3952-ddaf-4858-af83-458d1c01f0e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:00.069875 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:00.069795 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39fe3952-ddaf-4858-af83-458d1c01f0e8-istiod-ca-cert" pod "39fe3952-ddaf-4858-af83-458d1c01f0e8" (UID: "39fe3952-ddaf-4858-af83-458d1c01f0e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:00.069875 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:00.069820 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39fe3952-ddaf-4858-af83-458d1c01f0e8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39fe3952-ddaf-4858-af83-458d1c01f0e8-istio-envoy\") pod \"39fe3952-ddaf-4858-af83-458d1c01f0e8\" (UID: \"39fe3952-ddaf-4858-af83-458d1c01f0e8\") " Oct 30 21:54:00.070074 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:00.069923 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39fe3952-ddaf-4858-af83-458d1c01f0e8-istio-data" pod "39fe3952-ddaf-4858-af83-458d1c01f0e8" (UID: "39fe3952-ddaf-4858-af83-458d1c01f0e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:00.070074 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:00.069984 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39fe3952-ddaf-4858-af83-458d1c01f0e8-workload-socket" pod "39fe3952-ddaf-4858-af83-458d1c01f0e8" (UID: "39fe3952-ddaf-4858-af83-458d1c01f0e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:00.070392 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:00.070371 5748 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39fe3952-ddaf-4858-af83-458d1c01f0e8-credential-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:00.070392 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:00.070391 5748 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39fe3952-ddaf-4858-af83-458d1c01f0e8-istiod-ca-cert\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:00.070570 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:00.070400 5748 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39fe3952-ddaf-4858-af83-458d1c01f0e8-workload-certs\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:00.070570 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:00.070409 5748 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39fe3952-ddaf-4858-af83-458d1c01f0e8-istio-data\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:00.070570 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:00.070422 5748 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39fe3952-ddaf-4858-af83-458d1c01f0e8-workload-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:00.071961 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:00.071932 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39fe3952-ddaf-4858-af83-458d1c01f0e8-istio-token" pod "39fe3952-ddaf-4858-af83-458d1c01f0e8" (UID: "39fe3952-ddaf-4858-af83-458d1c01f0e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:00.072067 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:00.071999 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39fe3952-ddaf-4858-af83-458d1c01f0e8-istio-envoy" pod "39fe3952-ddaf-4858-af83-458d1c01f0e8" (UID: "39fe3952-ddaf-4858-af83-458d1c01f0e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:00.072167 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:00.072148 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39fe3952-ddaf-4858-af83-458d1c01f0e8-istio-podinfo" pod "39fe3952-ddaf-4858-af83-458d1c01f0e8" (UID: "39fe3952-ddaf-4858-af83-458d1c01f0e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:54:00.072332 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:00.072310 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39fe3952-ddaf-4858-af83-458d1c01f0e8-kube-api-access-tqc2m" pod "39fe3952-ddaf-4858-af83-458d1c01f0e8" (UID: "39fe3952-ddaf-4858-af83-458d1c01f0e8"). InnerVolumeSpecName "kube-api-access-tqc2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:00.101622 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:00.101594 5748 generic.go:356] "Generic (PLEG): container finished" podID="39fe3952-ddaf-4858-af83-458d1c01f0e8" containerID="b3989391e8071317375a147526b7af75d2101b4587f0de7ec89098f2d587b0bf" exitCode=0 Oct 30 21:54:00.101744 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:00.101633 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m" event={"ID":"39fe3952-ddaf-4858-af83-458d1c01f0e8","Type":"ContainerDied","Data":"b3989391e8071317375a147526b7af75d2101b4587f0de7ec89098f2d587b0bf"} Oct 30 21:54:00.101744 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:00.101647 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m" Oct 30 21:54:00.101744 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:00.101699 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m" event={"ID":"39fe3952-ddaf-4858-af83-458d1c01f0e8","Type":"ContainerDied","Data":"74c73d0a805e64b79599f0262ead880992de351e0aea7a5b3566d355e1d5dbcf"} Oct 30 21:54:00.101744 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:00.101720 5748 scope.go:122] "RemoveContainer" containerID="b3989391e8071317375a147526b7af75d2101b4587f0de7ec89098f2d587b0bf" Oct 30 21:54:00.112510 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:00.112493 5748 scope.go:122] "RemoveContainer" containerID="b3989391e8071317375a147526b7af75d2101b4587f0de7ec89098f2d587b0bf" Oct 30 21:54:00.112782 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:54:00.112758 5748 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3989391e8071317375a147526b7af75d2101b4587f0de7ec89098f2d587b0bf\": container with ID starting with b3989391e8071317375a147526b7af75d2101b4587f0de7ec89098f2d587b0bf not found: ID does not exist" containerID="b3989391e8071317375a147526b7af75d2101b4587f0de7ec89098f2d587b0bf" Oct 30 21:54:00.112878 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:00.112788 5748 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3989391e8071317375a147526b7af75d2101b4587f0de7ec89098f2d587b0bf"} err="failed to get container status \"b3989391e8071317375a147526b7af75d2101b4587f0de7ec89098f2d587b0bf\": rpc error: code = NotFound desc = could not find container \"b3989391e8071317375a147526b7af75d2101b4587f0de7ec89098f2d587b0bf\": container with ID starting with b3989391e8071317375a147526b7af75d2101b4587f0de7ec89098f2d587b0bf not found: ID does not exist" Oct 30 21:54:00.144734 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:54:00.144706 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:54:00.171492 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:00.171469 5748 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39fe3952-ddaf-4858-af83-458d1c01f0e8-istio-token\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:00.171492 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:00.171493 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqc2m\" (UniqueName: \"kubernetes.io/projected/39fe3952-ddaf-4858-af83-458d1c01f0e8-kube-api-access-tqc2m\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:00.171661 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:00.171504 5748 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39fe3952-ddaf-4858-af83-458d1c01f0e8-istio-envoy\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:00.171661 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:00.171512 5748 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39fe3952-ddaf-4858-af83-458d1c01f0e8-istio-podinfo\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:00.297287 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:00.297172 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m"] Oct 30 21:54:00.300798 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:00.300771 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-qxp5m"] Oct 30 21:54:00.419623 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:00.419590 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39fe3952-ddaf-4858-af83-458d1c01f0e8" path="/var/lib/kubelet/pods/39fe3952-ddaf-4858-af83-458d1c01f0e8/volumes" Oct 30 21:54:02.237707 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:02.237669 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh"] Oct 30 21:54:02.238667 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:02.238642 5748 cpu_manager.go:414] "containerMap: removing container" podUID="39fe3952-ddaf-4858-af83-458d1c01f0e8" containerName="istio-proxy" Oct 30 21:54:02.238667 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:02.238667 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39fe3952-ddaf-4858-af83-458d1c01f0e8" containerName="istio-proxy" Oct 30 21:54:02.238876 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:02.238812 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="39fe3952-ddaf-4858-af83-458d1c01f0e8" containerName="istio-proxy" Oct 30 21:54:02.244748 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:02.244723 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh" Oct 30 21:54:02.307380 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:02.307351 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-e51c52-openshift-default-dockercfg-5h929\"" Oct 30 21:54:02.308047 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:02.308021 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3d0a7fa-e727-4776-b3c2-db1218760332-istio-envoy\") pod \"e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh\" (UID: \"f3d0a7fa-e727-4776-b3c2-db1218760332\") " pod="openshift-ingress/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh" Oct 30 21:54:02.308210 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:02.308100 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3d0a7fa-e727-4776-b3c2-db1218760332-istio-data\") pod \"e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh\" (UID: \"f3d0a7fa-e727-4776-b3c2-db1218760332\") " pod="openshift-ingress/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh" Oct 30 21:54:02.308210 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:02.308138 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5klvr\" (UniqueName: \"kubernetes.io/projected/f3d0a7fa-e727-4776-b3c2-db1218760332-kube-api-access-5klvr\") pod \"e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh\" (UID: \"f3d0a7fa-e727-4776-b3c2-db1218760332\") " pod="openshift-ingress/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh" Oct 30 21:54:02.308210 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:02.308168 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3d0a7fa-e727-4776-b3c2-db1218760332-istio-podinfo\") pod \"e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh\" (UID: \"f3d0a7fa-e727-4776-b3c2-db1218760332\") " pod="openshift-ingress/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh" Oct 30 21:54:02.308210 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:02.308200 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3d0a7fa-e727-4776-b3c2-db1218760332-istiod-ca-cert\") pod \"e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh\" (UID: \"f3d0a7fa-e727-4776-b3c2-db1218760332\") " pod="openshift-ingress/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh" Oct 30 21:54:02.308413 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:02.308232 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d0a7fa-e727-4776-b3c2-db1218760332-credential-socket\") pod \"e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh\" (UID: \"f3d0a7fa-e727-4776-b3c2-db1218760332\") " pod="openshift-ingress/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh" Oct 30 21:54:02.308413 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:02.308273 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3d0a7fa-e727-4776-b3c2-db1218760332-workload-certs\") pod \"e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh\" (UID: \"f3d0a7fa-e727-4776-b3c2-db1218760332\") " pod="openshift-ingress/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh" Oct 30 21:54:02.308575 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:02.308466 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d0a7fa-e727-4776-b3c2-db1218760332-workload-socket\") pod \"e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh\" (UID: \"f3d0a7fa-e727-4776-b3c2-db1218760332\") " pod="openshift-ingress/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh" Oct 30 21:54:02.308575 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:02.308501 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3d0a7fa-e727-4776-b3c2-db1218760332-istio-token\") pod \"e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh\" (UID: \"f3d0a7fa-e727-4776-b3c2-db1218760332\") " pod="openshift-ingress/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh" Oct 30 21:54:02.367050 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:02.366871 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh"] Oct 30 21:54:02.419694 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:02.419647 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d0a7fa-e727-4776-b3c2-db1218760332-credential-socket\") pod \"e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh\" (UID: \"f3d0a7fa-e727-4776-b3c2-db1218760332\") " pod="openshift-ingress/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh" Oct 30 21:54:02.423961 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:02.422630 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3d0a7fa-e727-4776-b3c2-db1218760332-workload-certs\") pod \"e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh\" (UID: \"f3d0a7fa-e727-4776-b3c2-db1218760332\") " pod="openshift-ingress/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh" Oct 30 21:54:02.423961 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:02.422852 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d0a7fa-e727-4776-b3c2-db1218760332-workload-socket\") pod \"e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh\" (UID: \"f3d0a7fa-e727-4776-b3c2-db1218760332\") " pod="openshift-ingress/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh" Oct 30 21:54:02.423961 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:02.422891 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3d0a7fa-e727-4776-b3c2-db1218760332-istio-token\") pod \"e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh\" (UID: \"f3d0a7fa-e727-4776-b3c2-db1218760332\") " pod="openshift-ingress/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh" Oct 30 21:54:02.423961 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:02.423066 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3d0a7fa-e727-4776-b3c2-db1218760332-istio-envoy\") pod \"e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh\" (UID: \"f3d0a7fa-e727-4776-b3c2-db1218760332\") " pod="openshift-ingress/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh" Oct 30 21:54:02.423961 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:02.423148 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3d0a7fa-e727-4776-b3c2-db1218760332-istio-data\") pod \"e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh\" (UID: \"f3d0a7fa-e727-4776-b3c2-db1218760332\") " pod="openshift-ingress/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh" Oct 30 21:54:02.423961 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:02.423181 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5klvr\" (UniqueName: \"kubernetes.io/projected/f3d0a7fa-e727-4776-b3c2-db1218760332-kube-api-access-5klvr\") pod \"e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh\" (UID: \"f3d0a7fa-e727-4776-b3c2-db1218760332\") " pod="openshift-ingress/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh" Oct 30 21:54:02.423961 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:02.423211 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3d0a7fa-e727-4776-b3c2-db1218760332-istio-podinfo\") pod \"e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh\" (UID: \"f3d0a7fa-e727-4776-b3c2-db1218760332\") " pod="openshift-ingress/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh" Oct 30 21:54:02.423961 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:02.423244 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3d0a7fa-e727-4776-b3c2-db1218760332-istiod-ca-cert\") pod \"e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh\" (UID: \"f3d0a7fa-e727-4776-b3c2-db1218760332\") " pod="openshift-ingress/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh" Oct 30 21:54:02.424490 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:02.424093 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3d0a7fa-e727-4776-b3c2-db1218760332-istiod-ca-cert\") pod \"e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh\" (UID: \"f3d0a7fa-e727-4776-b3c2-db1218760332\") " pod="openshift-ingress/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh" Oct 30 21:54:02.424490 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:02.420265 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d0a7fa-e727-4776-b3c2-db1218760332-credential-socket\") pod \"e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh\" (UID: \"f3d0a7fa-e727-4776-b3c2-db1218760332\") " pod="openshift-ingress/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh" Oct 30 21:54:02.424610 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:02.424577 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3d0a7fa-e727-4776-b3c2-db1218760332-workload-certs\") pod \"e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh\" (UID: \"f3d0a7fa-e727-4776-b3c2-db1218760332\") " pod="openshift-ingress/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh" Oct 30 21:54:02.425016 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:02.424949 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3d0a7fa-e727-4776-b3c2-db1218760332-istio-data\") pod \"e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh\" (UID: \"f3d0a7fa-e727-4776-b3c2-db1218760332\") " pod="openshift-ingress/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh" Oct 30 21:54:02.425206 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:02.425188 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d0a7fa-e727-4776-b3c2-db1218760332-workload-socket\") pod \"e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh\" (UID: \"f3d0a7fa-e727-4776-b3c2-db1218760332\") " pod="openshift-ingress/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh" Oct 30 21:54:02.430882 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:02.430808 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3d0a7fa-e727-4776-b3c2-db1218760332-istio-podinfo\") pod \"e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh\" (UID: \"f3d0a7fa-e727-4776-b3c2-db1218760332\") " pod="openshift-ingress/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh" Oct 30 21:54:02.435474 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:02.435447 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3d0a7fa-e727-4776-b3c2-db1218760332-istio-envoy\") pod \"e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh\" (UID: \"f3d0a7fa-e727-4776-b3c2-db1218760332\") " pod="openshift-ingress/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh" Oct 30 21:54:02.486426 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:02.486360 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3d0a7fa-e727-4776-b3c2-db1218760332-istio-token\") pod \"e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh\" (UID: \"f3d0a7fa-e727-4776-b3c2-db1218760332\") " pod="openshift-ingress/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh" Oct 30 21:54:02.487465 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:02.487436 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5klvr\" (UniqueName: \"kubernetes.io/projected/f3d0a7fa-e727-4776-b3c2-db1218760332-kube-api-access-5klvr\") pod \"e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh\" (UID: \"f3d0a7fa-e727-4776-b3c2-db1218760332\") " pod="openshift-ingress/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh" Oct 30 21:54:02.569286 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:02.569195 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh" Oct 30 21:54:03.026085 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:54:03.026048 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3d0a7fa_e727_4776_b3c2_db1218760332.slice/crio-d32bfb19244f10ea6c5d4f03015f28cf84e4b91f4408c35beb3e302de40c7c99 WatchSource:0}: Error finding container d32bfb19244f10ea6c5d4f03015f28cf84e4b91f4408c35beb3e302de40c7c99: Status 404 returned error can't find the container with id d32bfb19244f10ea6c5d4f03015f28cf84e4b91f4408c35beb3e302de40c7c99 Oct 30 21:54:03.026371 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:03.026326 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh"] Oct 30 21:54:03.030639 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:03.030601 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:54:03.030770 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:03.030684 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:54:03.030770 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:03.030729 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:54:03.125909 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:03.125856 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh" event={"ID":"f3d0a7fa-e727-4776-b3c2-db1218760332","Type":"ContainerStarted","Data":"eaa55b38e0c967494780c31fd90097d9a1a29c32f3bfc7f22c3245291dbda6ca"} Oct 30 21:54:03.126075 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:03.125940 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh" event={"ID":"f3d0a7fa-e727-4776-b3c2-db1218760332","Type":"ContainerStarted","Data":"d32bfb19244f10ea6c5d4f03015f28cf84e4b91f4408c35beb3e302de40c7c99"} Oct 30 21:54:03.255870 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:03.255815 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh" podStartSLOduration=1.255799948 podStartE2EDuration="1.255799948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:03.252592322 +0000 UTC m=+1826.995113253" watchObservedRunningTime="2026-10-30 21:54:03.255799948 +0000 UTC m=+1826.998320884" Oct 30 21:54:03.569967 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:03.569936 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh" Oct 30 21:54:04.575512 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:04.575483 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh" Oct 30 21:54:05.138557 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:05.138528 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh" Oct 30 21:54:05.139542 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:05.139518 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh" Oct 30 21:54:17.051198 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:17.050308 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4259d052-phh64"] Oct 30 21:54:17.100031 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:17.100001 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4259d052-phh64"] Oct 30 21:54:18.431369 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:18.431329 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="889f25ea-9b78-4350-a0fb-770b4626a4e6" path="/var/lib/kubelet/pods/889f25ea-9b78-4350-a0fb-770b4626a4e6/volumes" Oct 30 21:54:23.918462 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:23.918417 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5"] Oct 30 21:54:23.923021 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:23.922996 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5" Oct 30 21:54:23.934129 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:23.934098 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph2m2\" (UniqueName: \"kubernetes.io/projected/1faeab78-65f0-489f-9aeb-d651dd7cac1b-kube-api-access-ph2m2\") pod \"e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5\" (UID: \"1faeab78-65f0-489f-9aeb-d651dd7cac1b\") " pod="openshift-ingress/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5" Oct 30 21:54:23.934277 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:23.934167 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1faeab78-65f0-489f-9aeb-d651dd7cac1b-istio-podinfo\") pod \"e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5\" (UID: \"1faeab78-65f0-489f-9aeb-d651dd7cac1b\") " pod="openshift-ingress/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5" Oct 30 21:54:23.934361 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:23.934294 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1faeab78-65f0-489f-9aeb-d651dd7cac1b-istio-envoy\") pod \"e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5\" (UID: \"1faeab78-65f0-489f-9aeb-d651dd7cac1b\") " pod="openshift-ingress/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5" Oct 30 21:54:23.934436 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:23.934415 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1faeab78-65f0-489f-9aeb-d651dd7cac1b-credential-socket\") pod \"e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5\" (UID: \"1faeab78-65f0-489f-9aeb-d651dd7cac1b\") " pod="openshift-ingress/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5" Oct 30 21:54:23.934571 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:23.934529 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1faeab78-65f0-489f-9aeb-d651dd7cac1b-istio-token\") pod \"e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5\" (UID: \"1faeab78-65f0-489f-9aeb-d651dd7cac1b\") " pod="openshift-ingress/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5" Oct 30 21:54:23.934652 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:23.934593 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1faeab78-65f0-489f-9aeb-d651dd7cac1b-istiod-ca-cert\") pod \"e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5\" (UID: \"1faeab78-65f0-489f-9aeb-d651dd7cac1b\") " pod="openshift-ingress/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5" Oct 30 21:54:23.934710 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:23.934660 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1faeab78-65f0-489f-9aeb-d651dd7cac1b-istio-data\") pod \"e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5\" (UID: \"1faeab78-65f0-489f-9aeb-d651dd7cac1b\") " pod="openshift-ingress/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5" Oct 30 21:54:23.934710 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:23.934702 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1faeab78-65f0-489f-9aeb-d651dd7cac1b-workload-certs\") pod \"e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5\" (UID: \"1faeab78-65f0-489f-9aeb-d651dd7cac1b\") " pod="openshift-ingress/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5" Oct 30 21:54:23.934793 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:23.934734 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1faeab78-65f0-489f-9aeb-d651dd7cac1b-workload-socket\") pod \"e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5\" (UID: \"1faeab78-65f0-489f-9aeb-d651dd7cac1b\") " pod="openshift-ingress/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5" Oct 30 21:54:23.979740 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:23.979704 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-78c632-openshift-default-dockercfg-c4hbn\"" Oct 30 21:54:24.035635 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:24.035601 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5"] Oct 30 21:54:24.035950 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:24.035888 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1faeab78-65f0-489f-9aeb-d651dd7cac1b-istiod-ca-cert\") pod \"e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5\" (UID: \"1faeab78-65f0-489f-9aeb-d651dd7cac1b\") " pod="openshift-ingress/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5" Oct 30 21:54:24.036101 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:24.036005 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1faeab78-65f0-489f-9aeb-d651dd7cac1b-istio-data\") pod \"e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5\" (UID: \"1faeab78-65f0-489f-9aeb-d651dd7cac1b\") " pod="openshift-ingress/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5" Oct 30 21:54:24.036101 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:24.036050 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1faeab78-65f0-489f-9aeb-d651dd7cac1b-workload-certs\") pod \"e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5\" (UID: \"1faeab78-65f0-489f-9aeb-d651dd7cac1b\") " pod="openshift-ingress/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5" Oct 30 21:54:24.036229 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:24.036193 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1faeab78-65f0-489f-9aeb-d651dd7cac1b-workload-socket\") pod \"e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5\" (UID: \"1faeab78-65f0-489f-9aeb-d651dd7cac1b\") " pod="openshift-ingress/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5" Oct 30 21:54:24.036322 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:24.036302 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph2m2\" (UniqueName: \"kubernetes.io/projected/1faeab78-65f0-489f-9aeb-d651dd7cac1b-kube-api-access-ph2m2\") pod \"e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5\" (UID: \"1faeab78-65f0-489f-9aeb-d651dd7cac1b\") " pod="openshift-ingress/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5" Oct 30 21:54:24.036396 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:24.036371 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1faeab78-65f0-489f-9aeb-d651dd7cac1b-istio-podinfo\") pod \"e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5\" (UID: \"1faeab78-65f0-489f-9aeb-d651dd7cac1b\") " pod="openshift-ingress/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5" Oct 30 21:54:24.036461 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:24.036421 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1faeab78-65f0-489f-9aeb-d651dd7cac1b-istio-envoy\") pod \"e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5\" (UID: \"1faeab78-65f0-489f-9aeb-d651dd7cac1b\") " pod="openshift-ingress/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5" Oct 30 21:54:24.036524 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:24.036472 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1faeab78-65f0-489f-9aeb-d651dd7cac1b-istio-data\") pod \"e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5\" (UID: \"1faeab78-65f0-489f-9aeb-d651dd7cac1b\") " pod="openshift-ingress/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5" Oct 30 21:54:24.036582 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:24.036525 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1faeab78-65f0-489f-9aeb-d651dd7cac1b-credential-socket\") pod \"e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5\" (UID: \"1faeab78-65f0-489f-9aeb-d651dd7cac1b\") " pod="openshift-ingress/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5" Oct 30 21:54:24.036582 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:24.036559 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1faeab78-65f0-489f-9aeb-d651dd7cac1b-workload-socket\") pod \"e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5\" (UID: \"1faeab78-65f0-489f-9aeb-d651dd7cac1b\") " pod="openshift-ingress/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5" Oct 30 21:54:24.036698 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:24.036657 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1faeab78-65f0-489f-9aeb-d651dd7cac1b-istio-token\") pod \"e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5\" (UID: \"1faeab78-65f0-489f-9aeb-d651dd7cac1b\") " pod="openshift-ingress/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5" Oct 30 21:54:24.036698 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:24.036668 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1faeab78-65f0-489f-9aeb-d651dd7cac1b-istiod-ca-cert\") pod \"e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5\" (UID: \"1faeab78-65f0-489f-9aeb-d651dd7cac1b\") " pod="openshift-ingress/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5" Oct 30 21:54:24.036865 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:24.036840 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1faeab78-65f0-489f-9aeb-d651dd7cac1b-credential-socket\") pod \"e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5\" (UID: \"1faeab78-65f0-489f-9aeb-d651dd7cac1b\") " pod="openshift-ingress/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5" Oct 30 21:54:24.037253 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:24.037227 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1faeab78-65f0-489f-9aeb-d651dd7cac1b-workload-certs\") pod \"e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5\" (UID: \"1faeab78-65f0-489f-9aeb-d651dd7cac1b\") " pod="openshift-ingress/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5" Oct 30 21:54:24.039285 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:24.039257 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1faeab78-65f0-489f-9aeb-d651dd7cac1b-istio-envoy\") pod \"e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5\" (UID: \"1faeab78-65f0-489f-9aeb-d651dd7cac1b\") " pod="openshift-ingress/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5" Oct 30 21:54:24.039403 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:24.039304 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1faeab78-65f0-489f-9aeb-d651dd7cac1b-istio-podinfo\") pod \"e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5\" (UID: \"1faeab78-65f0-489f-9aeb-d651dd7cac1b\") " pod="openshift-ingress/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5" Oct 30 21:54:24.099098 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:24.099063 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph2m2\" (UniqueName: \"kubernetes.io/projected/1faeab78-65f0-489f-9aeb-d651dd7cac1b-kube-api-access-ph2m2\") pod \"e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5\" (UID: \"1faeab78-65f0-489f-9aeb-d651dd7cac1b\") " pod="openshift-ingress/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5" Oct 30 21:54:24.099525 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:24.099501 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1faeab78-65f0-489f-9aeb-d651dd7cac1b-istio-token\") pod \"e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5\" (UID: \"1faeab78-65f0-489f-9aeb-d651dd7cac1b\") " pod="openshift-ingress/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5" Oct 30 21:54:24.239213 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:24.239123 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5" Oct 30 21:54:24.678994 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:24.678966 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5"] Oct 30 21:54:24.681155 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:54:24.681125 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1faeab78_65f0_489f_9aeb_d651dd7cac1b.slice/crio-84ad385b0c19d90ea248f93fc2c6e7656157dc032d801ce21bf2ab493e024148 WatchSource:0}: Error finding container 84ad385b0c19d90ea248f93fc2c6e7656157dc032d801ce21bf2ab493e024148: Status 404 returned error can't find the container with id 84ad385b0c19d90ea248f93fc2c6e7656157dc032d801ce21bf2ab493e024148 Oct 30 21:54:24.683851 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:24.683814 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:54:24.683982 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:24.683924 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:54:24.683982 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:24.683973 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:54:25.280744 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:25.280705 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5" event={"ID":"1faeab78-65f0-489f-9aeb-d651dd7cac1b","Type":"ContainerStarted","Data":"afa4d973d6d47fbdd9c9b149a638994e3606321fc430e0e577c5e9ab84c942c9"} Oct 30 21:54:25.281193 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:25.280765 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5" event={"ID":"1faeab78-65f0-489f-9aeb-d651dd7cac1b","Type":"ContainerStarted","Data":"84ad385b0c19d90ea248f93fc2c6e7656157dc032d801ce21bf2ab493e024148"} Oct 30 21:54:25.407628 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:25.407563 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5" podStartSLOduration=2.407545781 podStartE2EDuration="2.407545781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:25.402682381 +0000 UTC m=+1849.145203315" watchObservedRunningTime="2026-10-30 21:54:25.407545781 +0000 UTC m=+1849.150066724" Oct 30 21:54:26.242953 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:26.239833 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5" Oct 30 21:54:26.247308 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:26.247271 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5" Oct 30 21:54:26.288315 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:26.288275 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5" Oct 30 21:54:26.289134 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:26.289117 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5" Oct 30 21:54:30.146442 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:54:30.146398 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:54:37.562984 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:37.562951 5748 scope.go:122] "RemoveContainer" containerID="8c6f26560cb5d014e8998ebe9c2cfdf8c33df29c9e5ff1342ed3e68db4f716e6" Oct 30 21:54:37.573503 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:37.573481 5748 scope.go:122] "RemoveContainer" containerID="8a52594d3977ba0af3dca9271c4e34e7ce265ff7538f7da7d18c752c3a2884c9" Oct 30 21:54:37.582471 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:37.582446 5748 scope.go:122] "RemoveContainer" containerID="5939d8046c041a2205e70b77341d4eaaf8deefbdcfa135756a890253f2515a4d" Oct 30 21:54:37.590808 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:37.590787 5748 scope.go:122] "RemoveContainer" containerID="0909e3ac6b39bac974b51c83159a79980709ca5835787da976a920513f3a2608" Oct 30 21:54:42.137994 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:42.137952 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-78c632-7784567c9b-bm6kt"] Oct 30 21:54:42.145470 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:42.145447 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-78c632-7784567c9b-bm6kt" Oct 30 21:54:42.195633 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:42.190535 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-78c632-7784567c9b-bm6kt"] Oct 30 21:54:42.209739 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:42.209700 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/950134bb-c2f9-4de9-a6a2-366a8e30f1b0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-78c632-7784567c9b-bm6kt\" (UID: \"950134bb-c2f9-4de9-a6a2-366a8e30f1b0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-78c632-7784567c9b-bm6kt" Oct 30 21:54:42.209967 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:42.209808 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgrpr\" (UniqueName: \"kubernetes.io/projected/950134bb-c2f9-4de9-a6a2-366a8e30f1b0-kube-api-access-fgrpr\") pod \"payload-processing-e2e-shared-b-w4-78c632-7784567c9b-bm6kt\" (UID: \"950134bb-c2f9-4de9-a6a2-366a8e30f1b0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-78c632-7784567c9b-bm6kt" Oct 30 21:54:42.213291 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:42.213255 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-78c632\"" Oct 30 21:54:42.213584 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:42.213547 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-78c632-dockercfg-7bdw5\"" Oct 30 21:54:42.311696 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:42.311658 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/950134bb-c2f9-4de9-a6a2-366a8e30f1b0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-78c632-7784567c9b-bm6kt\" (UID: \"950134bb-c2f9-4de9-a6a2-366a8e30f1b0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-78c632-7784567c9b-bm6kt" Oct 30 21:54:42.311921 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:42.311740 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgrpr\" (UniqueName: \"kubernetes.io/projected/950134bb-c2f9-4de9-a6a2-366a8e30f1b0-kube-api-access-fgrpr\") pod \"payload-processing-e2e-shared-b-w4-78c632-7784567c9b-bm6kt\" (UID: \"950134bb-c2f9-4de9-a6a2-366a8e30f1b0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-78c632-7784567c9b-bm6kt" Oct 30 21:54:42.312600 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:42.312568 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/950134bb-c2f9-4de9-a6a2-366a8e30f1b0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-78c632-7784567c9b-bm6kt\" (UID: \"950134bb-c2f9-4de9-a6a2-366a8e30f1b0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-78c632-7784567c9b-bm6kt" Oct 30 21:54:42.379538 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:42.379507 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgrpr\" (UniqueName: \"kubernetes.io/projected/950134bb-c2f9-4de9-a6a2-366a8e30f1b0-kube-api-access-fgrpr\") pod \"payload-processing-e2e-shared-b-w4-78c632-7784567c9b-bm6kt\" (UID: \"950134bb-c2f9-4de9-a6a2-366a8e30f1b0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-78c632-7784567c9b-bm6kt" Oct 30 21:54:42.502698 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:42.502621 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-78c632-7784567c9b-bm6kt" Oct 30 21:54:42.886161 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:42.886134 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-78c632-7784567c9b-bm6kt"] Oct 30 21:54:42.888107 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:54:42.888069 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod950134bb_c2f9_4de9_a6a2_366a8e30f1b0.slice/crio-20bbb7b6f9bb1dce9249d1f77e544dff5bce96ff48ae743417c3aaf73054136c WatchSource:0}: Error finding container 20bbb7b6f9bb1dce9249d1f77e544dff5bce96ff48ae743417c3aaf73054136c: Status 404 returned error can't find the container with id 20bbb7b6f9bb1dce9249d1f77e544dff5bce96ff48ae743417c3aaf73054136c Oct 30 21:54:43.426469 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:43.426427 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-78c632-7784567c9b-bm6kt" event={"ID":"950134bb-c2f9-4de9-a6a2-366a8e30f1b0","Type":"ContainerStarted","Data":"5381fb88994223f2889d6f45110f7668f5dc4000340e90cf34763599f4e3f2ee"} Oct 30 21:54:43.426963 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:43.426490 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-78c632-7784567c9b-bm6kt" event={"ID":"950134bb-c2f9-4de9-a6a2-366a8e30f1b0","Type":"ContainerStarted","Data":"20bbb7b6f9bb1dce9249d1f77e544dff5bce96ff48ae743417c3aaf73054136c"} Oct 30 21:54:43.426963 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:43.426616 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-78c632-7784567c9b-bm6kt" Oct 30 21:54:43.583542 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:43.583482 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-78c632-7784567c9b-bm6kt" podStartSLOduration=1.583466794 podStartE2EDuration="1.583466794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:43.581786573 +0000 UTC m=+1867.324307509" watchObservedRunningTime="2026-10-30 21:54:43.583466794 +0000 UTC m=+1867.325987728" Oct 30 21:54:50.188159 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:50.188122 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8709fb3-zffc6"] Oct 30 21:54:50.192512 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:50.192496 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8709fb3-zffc6" Oct 30 21:54:50.246623 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:50.246432 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:54:50.296829 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:50.296798 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:54:50.297063 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:50.297044 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 21:54:50.297124 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:50.297062 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:54:50.300198 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:50.300177 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8709fb3-zffc6"] Oct 30 21:54:50.326361 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:50.326335 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/243e1423-410b-4e99-88d5-59010335385a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-b8709fb3-zffc6\" (UID: \"243e1423-410b-4e99-88d5-59010335385a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8709fb3-zffc6" Oct 30 21:54:50.326503 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:50.326369 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts2ps\" (UniqueName: \"kubernetes.io/projected/243e1423-410b-4e99-88d5-59010335385a-kube-api-access-ts2ps\") pod \"maas-api-revoke-keys-e2e-mt-b8709fb3-zffc6\" (UID: \"243e1423-410b-4e99-88d5-59010335385a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8709fb3-zffc6" Oct 30 21:54:50.326657 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:50.326643 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/243e1423-410b-4e99-88d5-59010335385a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-b8709fb3-zffc6\" (UID: \"243e1423-410b-4e99-88d5-59010335385a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8709fb3-zffc6" Oct 30 21:54:50.427609 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:50.427574 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/243e1423-410b-4e99-88d5-59010335385a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-b8709fb3-zffc6\" (UID: \"243e1423-410b-4e99-88d5-59010335385a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8709fb3-zffc6" Oct 30 21:54:50.427769 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:50.427621 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/243e1423-410b-4e99-88d5-59010335385a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-b8709fb3-zffc6\" (UID: \"243e1423-410b-4e99-88d5-59010335385a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8709fb3-zffc6" Oct 30 21:54:50.427769 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:50.427669 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts2ps\" (UniqueName: \"kubernetes.io/projected/243e1423-410b-4e99-88d5-59010335385a-kube-api-access-ts2ps\") pod \"maas-api-revoke-keys-e2e-mt-b8709fb3-zffc6\" (UID: \"243e1423-410b-4e99-88d5-59010335385a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8709fb3-zffc6" Oct 30 21:54:50.428223 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:50.428202 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/243e1423-410b-4e99-88d5-59010335385a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-b8709fb3-zffc6\" (UID: \"243e1423-410b-4e99-88d5-59010335385a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8709fb3-zffc6" Oct 30 21:54:50.429943 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:50.429883 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/243e1423-410b-4e99-88d5-59010335385a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-b8709fb3-zffc6\" (UID: \"243e1423-410b-4e99-88d5-59010335385a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8709fb3-zffc6" Oct 30 21:54:50.490479 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:50.490413 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts2ps\" (UniqueName: \"kubernetes.io/projected/243e1423-410b-4e99-88d5-59010335385a-kube-api-access-ts2ps\") pod \"maas-api-revoke-keys-e2e-mt-b8709fb3-zffc6\" (UID: \"243e1423-410b-4e99-88d5-59010335385a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8709fb3-zffc6" Oct 30 21:54:50.506220 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:50.506192 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8709fb3-zffc6" Oct 30 21:54:50.883655 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:50.883579 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8709fb3-zffc6"] Oct 30 21:54:50.887653 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:54:50.887621 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod243e1423_410b_4e99_88d5_59010335385a.slice/crio-27729167b8f8fbcb030a008ba5a5a6ac3f794a10516917ff1febd3995ba990b4 WatchSource:0}: Error finding container 27729167b8f8fbcb030a008ba5a5a6ac3f794a10516917ff1febd3995ba990b4: Status 404 returned error can't find the container with id 27729167b8f8fbcb030a008ba5a5a6ac3f794a10516917ff1febd3995ba990b4 Oct 30 21:54:51.479597 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:51.479447 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8709fb3-zffc6" event={"ID":"243e1423-410b-4e99-88d5-59010335385a","Type":"ContainerStarted","Data":"317085ad620292f235705cd0094cd09a56dbb16ec4f4225c6233ddfeb0389b41"} Oct 30 21:54:51.479597 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:51.479501 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8709fb3-zffc6" event={"ID":"243e1423-410b-4e99-88d5-59010335385a","Type":"ContainerStarted","Data":"27729167b8f8fbcb030a008ba5a5a6ac3f794a10516917ff1febd3995ba990b4"} Oct 30 21:54:51.598636 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:51.598586 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8709fb3-zffc6" podStartSLOduration=1.5985722390000001 podStartE2EDuration="1.598572239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:51.597307969 +0000 UTC m=+1875.339828911" watchObservedRunningTime="2026-10-30 21:54:51.598572239 +0000 UTC m=+1875.341093175" Oct 30 21:54:52.488629 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:52.488591 5748 generic.go:356] "Generic (PLEG): container finished" podID="243e1423-410b-4e99-88d5-59010335385a" containerID="317085ad620292f235705cd0094cd09a56dbb16ec4f4225c6233ddfeb0389b41" exitCode=0 Oct 30 21:54:52.489208 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:52.488647 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8709fb3-zffc6" event={"ID":"243e1423-410b-4e99-88d5-59010335385a","Type":"ContainerDied","Data":"317085ad620292f235705cd0094cd09a56dbb16ec4f4225c6233ddfeb0389b41"} Oct 30 21:54:53.665127 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:53.665104 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8709fb3-zffc6" Oct 30 21:54:53.773180 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:53.773118 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/243e1423-410b-4e99-88d5-59010335385a-kube-api-access-ts2ps\" (UniqueName: \"kubernetes.io/projected/243e1423-410b-4e99-88d5-59010335385a-kube-api-access-ts2ps\") pod \"243e1423-410b-4e99-88d5-59010335385a\" (UID: \"243e1423-410b-4e99-88d5-59010335385a\") " Oct 30 21:54:53.773180 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:53.773161 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/243e1423-410b-4e99-88d5-59010335385a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/243e1423-410b-4e99-88d5-59010335385a-maas-api-openshift-ca\") pod \"243e1423-410b-4e99-88d5-59010335385a\" (UID: \"243e1423-410b-4e99-88d5-59010335385a\") " Oct 30 21:54:53.773475 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:53.773452 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/243e1423-410b-4e99-88d5-59010335385a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/243e1423-410b-4e99-88d5-59010335385a-maas-api-xks-ca\") pod \"243e1423-410b-4e99-88d5-59010335385a\" (UID: \"243e1423-410b-4e99-88d5-59010335385a\") " Oct 30 21:54:53.773520 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:53.773481 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/243e1423-410b-4e99-88d5-59010335385a-maas-api-openshift-ca" pod "243e1423-410b-4e99-88d5-59010335385a" (UID: "243e1423-410b-4e99-88d5-59010335385a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:53.773863 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:53.773846 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/243e1423-410b-4e99-88d5-59010335385a-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:53.775246 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:53.775224 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/243e1423-410b-4e99-88d5-59010335385a-kube-api-access-ts2ps" pod "243e1423-410b-4e99-88d5-59010335385a" (UID: "243e1423-410b-4e99-88d5-59010335385a"). InnerVolumeSpecName "kube-api-access-ts2ps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:53.775473 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:53.775447 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/243e1423-410b-4e99-88d5-59010335385a-maas-api-xks-ca" pod "243e1423-410b-4e99-88d5-59010335385a" (UID: "243e1423-410b-4e99-88d5-59010335385a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:54:53.875794 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:53.875751 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/243e1423-410b-4e99-88d5-59010335385a-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:53.875794 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:53.875793 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts2ps\" (UniqueName: \"kubernetes.io/projected/243e1423-410b-4e99-88d5-59010335385a-kube-api-access-ts2ps\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:54.436795 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:54.436770 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-78c632-7784567c9b-bm6kt" Oct 30 21:54:54.504299 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:54.504261 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8709fb3-zffc6" event={"ID":"243e1423-410b-4e99-88d5-59010335385a","Type":"ContainerDied","Data":"27729167b8f8fbcb030a008ba5a5a6ac3f794a10516917ff1febd3995ba990b4"} Oct 30 21:54:54.504299 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:54.504297 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8709fb3-zffc6" Oct 30 21:54:54.504511 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:54:54.504318 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27729167b8f8fbcb030a008ba5a5a6ac3f794a10516917ff1febd3995ba990b4" Oct 30 21:55:00.151827 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:55:00.151785 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:55:06.601058 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:55:06.601026 5748 generic.go:356] "Generic (PLEG): container finished" podID="e9aab906-835e-4acf-b20f-ef27d8bd8568" containerID="36b317e896eca6413b3fcf645a68946f9e9ae4a601b45ca6f7be2a2c3504adfe" exitCode=0 Oct 30 21:55:06.601513 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:55:06.601101 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226" event={"ID":"e9aab906-835e-4acf-b20f-ef27d8bd8568","Type":"ContainerDied","Data":"36b317e896eca6413b3fcf645a68946f9e9ae4a601b45ca6f7be2a2c3504adfe"} Oct 30 21:55:06.601513 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:55:06.601457 5748 scope.go:122] "RemoveContainer" containerID="36b317e896eca6413b3fcf645a68946f9e9ae4a601b45ca6f7be2a2c3504adfe" Oct 30 21:55:06.603058 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:55:06.603031 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:55:06.603138 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:55:06.603097 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:55:06.603138 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:55:06.603130 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:55:07.631511 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:55:07.631474 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226" event={"ID":"e9aab906-835e-4acf-b20f-ef27d8bd8568","Type":"ContainerStarted","Data":"9f4a7086c80be63b67796c15a89a6eb8fd2dc9d59045928508cbf8418b4ae59b"} Oct 30 21:55:08.025170 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:55:08.025133 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226" Oct 30 21:55:08.025428 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:55:08.025402 5748 patch_prober.go:29] interesting pod/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 21:55:08.025499 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:55:08.025462 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226" podUID="e9aab906-835e-4acf-b20f-ef27d8bd8568" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 21:55:09.025304 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:55:09.025264 5748 patch_prober.go:29] interesting pod/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 21:55:09.025781 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:55:09.025323 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226" podUID="e9aab906-835e-4acf-b20f-ef27d8bd8568" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 21:55:10.024281 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:55:10.024240 5748 patch_prober.go:29] interesting pod/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 21:55:10.024616 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:55:10.024574 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226" podUID="e9aab906-835e-4acf-b20f-ef27d8bd8568" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 21:55:11.025278 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:55:11.025238 5748 patch_prober.go:29] interesting pod/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 21:55:11.025671 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:55:11.025311 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226" podUID="e9aab906-835e-4acf-b20f-ef27d8bd8568" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 21:55:12.025327 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:55:12.025246 5748 patch_prober.go:29] interesting pod/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 21:55:12.025327 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:55:12.025310 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226" podUID="e9aab906-835e-4acf-b20f-ef27d8bd8568" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 21:55:13.028701 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:55:13.028671 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226" Oct 30 21:55:13.029171 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:55:13.029134 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226" Oct 30 21:55:13.029880 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:55:13.029861 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226" Oct 30 21:55:30.145833 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:55:30.145789 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:55:37.631522 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:55:37.631493 5748 scope.go:122] "RemoveContainer" containerID="46b04822991af3fc25a0613fe30f493e9cf365996fd170509bfaa999e952276b" Oct 30 21:55:49.562291 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:55:49.562254 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71a3d788-6rwdh"] Oct 30 21:55:49.562928 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:55:49.562878 5748 cpu_manager.go:414] "containerMap: removing container" podUID="243e1423-410b-4e99-88d5-59010335385a" containerName="revoke-keys" Oct 30 21:55:49.562928 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:55:49.562909 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="243e1423-410b-4e99-88d5-59010335385a" containerName="revoke-keys" Oct 30 21:55:49.563033 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:55:49.563020 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="243e1423-410b-4e99-88d5-59010335385a" containerName="revoke-keys" Oct 30 21:55:49.565743 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:55:49.565727 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71a3d788-6rwdh" Oct 30 21:55:49.621416 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:55:49.621391 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:55:49.621714 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:55:49.621693 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:55:49.672054 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:55:49.672029 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 21:55:49.675271 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:55:49.675251 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71a3d788-6rwdh"] Oct 30 21:55:49.688694 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:55:49.688671 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e552129-6a1f-41a3-a8c0-3dd7512447a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-71a3d788-6rwdh\" (UID: \"2e552129-6a1f-41a3-a8c0-3dd7512447a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71a3d788-6rwdh" Oct 30 21:55:49.688765 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:55:49.688708 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e552129-6a1f-41a3-a8c0-3dd7512447a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-71a3d788-6rwdh\" (UID: \"2e552129-6a1f-41a3-a8c0-3dd7512447a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71a3d788-6rwdh" Oct 30 21:55:49.688824 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:55:49.688813 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xh4m\" (UniqueName: \"kubernetes.io/projected/2e552129-6a1f-41a3-a8c0-3dd7512447a5-kube-api-access-9xh4m\") pod \"maas-api-revoke-keys-e2e-mt-71a3d788-6rwdh\" (UID: \"2e552129-6a1f-41a3-a8c0-3dd7512447a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71a3d788-6rwdh" Oct 30 21:55:49.721951 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:55:49.721928 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:55:49.790385 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:55:49.790357 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e552129-6a1f-41a3-a8c0-3dd7512447a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-71a3d788-6rwdh\" (UID: \"2e552129-6a1f-41a3-a8c0-3dd7512447a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71a3d788-6rwdh" Oct 30 21:55:49.790546 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:55:49.790410 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e552129-6a1f-41a3-a8c0-3dd7512447a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-71a3d788-6rwdh\" (UID: \"2e552129-6a1f-41a3-a8c0-3dd7512447a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71a3d788-6rwdh" Oct 30 21:55:49.790546 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:55:49.790516 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xh4m\" (UniqueName: \"kubernetes.io/projected/2e552129-6a1f-41a3-a8c0-3dd7512447a5-kube-api-access-9xh4m\") pod \"maas-api-revoke-keys-e2e-mt-71a3d788-6rwdh\" (UID: \"2e552129-6a1f-41a3-a8c0-3dd7512447a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71a3d788-6rwdh" Oct 30 21:55:49.791114 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:55:49.791091 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e552129-6a1f-41a3-a8c0-3dd7512447a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-71a3d788-6rwdh\" (UID: \"2e552129-6a1f-41a3-a8c0-3dd7512447a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71a3d788-6rwdh" Oct 30 21:55:49.792709 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:55:49.792691 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e552129-6a1f-41a3-a8c0-3dd7512447a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-71a3d788-6rwdh\" (UID: \"2e552129-6a1f-41a3-a8c0-3dd7512447a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71a3d788-6rwdh" Oct 30 21:55:49.853271 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:55:49.853207 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xh4m\" (UniqueName: \"kubernetes.io/projected/2e552129-6a1f-41a3-a8c0-3dd7512447a5-kube-api-access-9xh4m\") pod \"maas-api-revoke-keys-e2e-mt-71a3d788-6rwdh\" (UID: \"2e552129-6a1f-41a3-a8c0-3dd7512447a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71a3d788-6rwdh" Oct 30 21:55:49.879066 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:55:49.879045 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71a3d788-6rwdh" Oct 30 21:55:50.255128 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:55:50.255098 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71a3d788-6rwdh"] Oct 30 21:55:50.256343 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:55:50.256316 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e552129_6a1f_41a3_a8c0_3dd7512447a5.slice/crio-c74b6222a98c69db83221031a2405664c1e51444d397ecaaad935310d1f41905 WatchSource:0}: Error finding container c74b6222a98c69db83221031a2405664c1e51444d397ecaaad935310d1f41905: Status 404 returned error can't find the container with id c74b6222a98c69db83221031a2405664c1e51444d397ecaaad935310d1f41905 Oct 30 21:55:50.964176 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:55:50.964141 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71a3d788-6rwdh" event={"ID":"2e552129-6a1f-41a3-a8c0-3dd7512447a5","Type":"ContainerStarted","Data":"b2b02a331d3d71175637572a702adf1d3c98b39834b837a02248679b5f8e09fd"} Oct 30 21:55:50.964582 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:55:50.964205 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71a3d788-6rwdh" event={"ID":"2e552129-6a1f-41a3-a8c0-3dd7512447a5","Type":"ContainerStarted","Data":"c74b6222a98c69db83221031a2405664c1e51444d397ecaaad935310d1f41905"} Oct 30 21:55:51.093517 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:55:51.093463 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71a3d788-6rwdh" podStartSLOduration=2.093449481 podStartE2EDuration="2.093449481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:51.091215672 +0000 UTC m=+1934.833736631" watchObservedRunningTime="2026-10-30 21:55:51.093449481 +0000 UTC m=+1934.835970476" Oct 30 21:55:51.971968 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:55:51.971928 5748 generic.go:356] "Generic (PLEG): container finished" podID="2e552129-6a1f-41a3-a8c0-3dd7512447a5" containerID="b2b02a331d3d71175637572a702adf1d3c98b39834b837a02248679b5f8e09fd" exitCode=0 Oct 30 21:55:51.972347 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:55:51.972003 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71a3d788-6rwdh" event={"ID":"2e552129-6a1f-41a3-a8c0-3dd7512447a5","Type":"ContainerDied","Data":"b2b02a331d3d71175637572a702adf1d3c98b39834b837a02248679b5f8e09fd"} Oct 30 21:55:53.128359 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:55:53.128331 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71a3d788-6rwdh" Oct 30 21:55:53.238719 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:55:53.238642 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e552129-6a1f-41a3-a8c0-3dd7512447a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e552129-6a1f-41a3-a8c0-3dd7512447a5-maas-api-xks-ca\") pod \"2e552129-6a1f-41a3-a8c0-3dd7512447a5\" (UID: \"2e552129-6a1f-41a3-a8c0-3dd7512447a5\") " Oct 30 21:55:53.238871 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:55:53.238766 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e552129-6a1f-41a3-a8c0-3dd7512447a5-kube-api-access-9xh4m\" (UniqueName: \"kubernetes.io/projected/2e552129-6a1f-41a3-a8c0-3dd7512447a5-kube-api-access-9xh4m\") pod \"2e552129-6a1f-41a3-a8c0-3dd7512447a5\" (UID: \"2e552129-6a1f-41a3-a8c0-3dd7512447a5\") " Oct 30 21:55:53.238871 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:55:53.238862 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e552129-6a1f-41a3-a8c0-3dd7512447a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e552129-6a1f-41a3-a8c0-3dd7512447a5-maas-api-openshift-ca\") pod \"2e552129-6a1f-41a3-a8c0-3dd7512447a5\" (UID: \"2e552129-6a1f-41a3-a8c0-3dd7512447a5\") " Oct 30 21:55:53.239226 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:55:53.239196 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e552129-6a1f-41a3-a8c0-3dd7512447a5-maas-api-openshift-ca" pod "2e552129-6a1f-41a3-a8c0-3dd7512447a5" (UID: "2e552129-6a1f-41a3-a8c0-3dd7512447a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:55:53.239516 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:55:53.239496 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e552129-6a1f-41a3-a8c0-3dd7512447a5-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:53.240826 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:55:53.240800 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e552129-6a1f-41a3-a8c0-3dd7512447a5-maas-api-xks-ca" pod "2e552129-6a1f-41a3-a8c0-3dd7512447a5" (UID: "2e552129-6a1f-41a3-a8c0-3dd7512447a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:55:53.240935 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:55:53.240879 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e552129-6a1f-41a3-a8c0-3dd7512447a5-kube-api-access-9xh4m" pod "2e552129-6a1f-41a3-a8c0-3dd7512447a5" (UID: "2e552129-6a1f-41a3-a8c0-3dd7512447a5"). InnerVolumeSpecName "kube-api-access-9xh4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:53.341060 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:55:53.341017 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xh4m\" (UniqueName: \"kubernetes.io/projected/2e552129-6a1f-41a3-a8c0-3dd7512447a5-kube-api-access-9xh4m\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:53.341060 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:55:53.341053 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e552129-6a1f-41a3-a8c0-3dd7512447a5-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:53.986289 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:55:53.986253 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71a3d788-6rwdh" Oct 30 21:55:53.986498 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:55:53.986292 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71a3d788-6rwdh" event={"ID":"2e552129-6a1f-41a3-a8c0-3dd7512447a5","Type":"ContainerDied","Data":"c74b6222a98c69db83221031a2405664c1e51444d397ecaaad935310d1f41905"} Oct 30 21:55:53.986498 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:55:53.986348 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c74b6222a98c69db83221031a2405664c1e51444d397ecaaad935310d1f41905" Oct 30 21:56:00.145521 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:56:00.145484 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:56:02.924439 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:02.924398 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-79e10a-rqj7g"] Oct 30 21:56:02.926084 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:02.926038 5748 cpu_manager.go:414] "containerMap: removing container" podUID="2e552129-6a1f-41a3-a8c0-3dd7512447a5" containerName="revoke-keys" Oct 30 21:56:02.926084 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:02.926064 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e552129-6a1f-41a3-a8c0-3dd7512447a5" containerName="revoke-keys" Oct 30 21:56:02.926375 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:02.926260 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e552129-6a1f-41a3-a8c0-3dd7512447a5" containerName="revoke-keys" Oct 30 21:56:02.934946 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:02.929424 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-79e10a-rqj7g" Oct 30 21:56:03.009979 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:03.009934 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85e4ecda-eb68-4c06-abb0-b6b8ea5db24c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-79e10a-rqj7g\" (UID: \"85e4ecda-eb68-4c06-abb0-b6b8ea5db24c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-79e10a-rqj7g" Oct 30 21:56:03.010198 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:03.010171 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85e4ecda-eb68-4c06-abb0-b6b8ea5db24c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-79e10a-rqj7g\" (UID: \"85e4ecda-eb68-4c06-abb0-b6b8ea5db24c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-79e10a-rqj7g" Oct 30 21:56:03.010355 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:03.010337 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5jkd\" (UniqueName: \"kubernetes.io/projected/85e4ecda-eb68-4c06-abb0-b6b8ea5db24c-kube-api-access-h5jkd\") pod \"maas-api-revoke-keys-e2e-api-a-79e10a-rqj7g\" (UID: \"85e4ecda-eb68-4c06-abb0-b6b8ea5db24c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-79e10a-rqj7g" Oct 30 21:56:03.036693 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:03.036666 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:56:03.036923 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:03.036858 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:56:03.037057 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:03.036930 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 21:56:03.037301 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:03.037264 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:56:03.043961 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:03.041822 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-79e10a-rqj7g"] Oct 30 21:56:03.111608 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:03.111573 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85e4ecda-eb68-4c06-abb0-b6b8ea5db24c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-79e10a-rqj7g\" (UID: \"85e4ecda-eb68-4c06-abb0-b6b8ea5db24c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-79e10a-rqj7g" Oct 30 21:56:03.111778 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:03.111666 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5jkd\" (UniqueName: \"kubernetes.io/projected/85e4ecda-eb68-4c06-abb0-b6b8ea5db24c-kube-api-access-h5jkd\") pod \"maas-api-revoke-keys-e2e-api-a-79e10a-rqj7g\" (UID: \"85e4ecda-eb68-4c06-abb0-b6b8ea5db24c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-79e10a-rqj7g" Oct 30 21:56:03.111778 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:03.111707 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85e4ecda-eb68-4c06-abb0-b6b8ea5db24c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-79e10a-rqj7g\" (UID: \"85e4ecda-eb68-4c06-abb0-b6b8ea5db24c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-79e10a-rqj7g" Oct 30 21:56:03.112499 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:03.112468 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85e4ecda-eb68-4c06-abb0-b6b8ea5db24c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-79e10a-rqj7g\" (UID: \"85e4ecda-eb68-4c06-abb0-b6b8ea5db24c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-79e10a-rqj7g" Oct 30 21:56:03.114547 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:03.114526 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85e4ecda-eb68-4c06-abb0-b6b8ea5db24c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-79e10a-rqj7g\" (UID: \"85e4ecda-eb68-4c06-abb0-b6b8ea5db24c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-79e10a-rqj7g" Oct 30 21:56:03.174212 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:03.174187 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5jkd\" (UniqueName: \"kubernetes.io/projected/85e4ecda-eb68-4c06-abb0-b6b8ea5db24c-kube-api-access-h5jkd\") pod \"maas-api-revoke-keys-e2e-api-a-79e10a-rqj7g\" (UID: \"85e4ecda-eb68-4c06-abb0-b6b8ea5db24c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-79e10a-rqj7g" Oct 30 21:56:03.251334 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:03.251269 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-79e10a-rqj7g" Oct 30 21:56:03.644197 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:03.644166 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-79e10a-rqj7g"] Oct 30 21:56:03.645756 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:56:03.645721 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85e4ecda_eb68_4c06_abb0_b6b8ea5db24c.slice/crio-fb1064fb8dcd42ab8e85603fc300f0eec067dc6459b01b9d0e1005730979ba88 WatchSource:0}: Error finding container fb1064fb8dcd42ab8e85603fc300f0eec067dc6459b01b9d0e1005730979ba88: Status 404 returned error can't find the container with id fb1064fb8dcd42ab8e85603fc300f0eec067dc6459b01b9d0e1005730979ba88 Oct 30 21:56:04.053629 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:04.053596 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-79e10a-rqj7g" event={"ID":"85e4ecda-eb68-4c06-abb0-b6b8ea5db24c","Type":"ContainerStarted","Data":"fb1064fb8dcd42ab8e85603fc300f0eec067dc6459b01b9d0e1005730979ba88"} Oct 30 21:56:05.061771 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:05.061686 5748 generic.go:356] "Generic (PLEG): container finished" podID="85e4ecda-eb68-4c06-abb0-b6b8ea5db24c" containerID="28a4be276fa5fa913ef9879a8529e628a1c63e8fb303eb149b0eee4f2a323a00" exitCode=0 Oct 30 21:56:05.061771 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:05.061759 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-79e10a-rqj7g" event={"ID":"85e4ecda-eb68-4c06-abb0-b6b8ea5db24c","Type":"ContainerDied","Data":"28a4be276fa5fa913ef9879a8529e628a1c63e8fb303eb149b0eee4f2a323a00"} Oct 30 21:56:06.271406 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:06.271372 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-79e10a-rqj7g" Oct 30 21:56:06.377968 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:06.373408 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85e4ecda-eb68-4c06-abb0-b6b8ea5db24c-kube-api-access-h5jkd\" (UniqueName: \"kubernetes.io/projected/85e4ecda-eb68-4c06-abb0-b6b8ea5db24c-kube-api-access-h5jkd\") pod \"85e4ecda-eb68-4c06-abb0-b6b8ea5db24c\" (UID: \"85e4ecda-eb68-4c06-abb0-b6b8ea5db24c\") " Oct 30 21:56:06.377968 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:06.373680 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85e4ecda-eb68-4c06-abb0-b6b8ea5db24c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85e4ecda-eb68-4c06-abb0-b6b8ea5db24c-maas-api-xks-ca\") pod \"85e4ecda-eb68-4c06-abb0-b6b8ea5db24c\" (UID: \"85e4ecda-eb68-4c06-abb0-b6b8ea5db24c\") " Oct 30 21:56:06.377968 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:06.373821 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85e4ecda-eb68-4c06-abb0-b6b8ea5db24c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85e4ecda-eb68-4c06-abb0-b6b8ea5db24c-maas-api-openshift-ca\") pod \"85e4ecda-eb68-4c06-abb0-b6b8ea5db24c\" (UID: \"85e4ecda-eb68-4c06-abb0-b6b8ea5db24c\") " Oct 30 21:56:06.384363 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:06.384322 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85e4ecda-eb68-4c06-abb0-b6b8ea5db24c-maas-api-openshift-ca" pod "85e4ecda-eb68-4c06-abb0-b6b8ea5db24c" (UID: "85e4ecda-eb68-4c06-abb0-b6b8ea5db24c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:06.437921 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:06.434167 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85e4ecda-eb68-4c06-abb0-b6b8ea5db24c-kube-api-access-h5jkd" pod "85e4ecda-eb68-4c06-abb0-b6b8ea5db24c" (UID: "85e4ecda-eb68-4c06-abb0-b6b8ea5db24c"). InnerVolumeSpecName "kube-api-access-h5jkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:06.463956 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:06.463395 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85e4ecda-eb68-4c06-abb0-b6b8ea5db24c-maas-api-xks-ca" pod "85e4ecda-eb68-4c06-abb0-b6b8ea5db24c" (UID: "85e4ecda-eb68-4c06-abb0-b6b8ea5db24c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:56:06.484583 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:06.484551 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85e4ecda-eb68-4c06-abb0-b6b8ea5db24c-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:06.484785 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:06.484769 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85e4ecda-eb68-4c06-abb0-b6b8ea5db24c-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:06.484969 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:06.484884 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5jkd\" (UniqueName: \"kubernetes.io/projected/85e4ecda-eb68-4c06-abb0-b6b8ea5db24c-kube-api-access-h5jkd\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:07.079422 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:07.079397 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-79e10a-rqj7g" Oct 30 21:56:07.079680 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:07.079392 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-79e10a-rqj7g" event={"ID":"85e4ecda-eb68-4c06-abb0-b6b8ea5db24c","Type":"ContainerDied","Data":"fb1064fb8dcd42ab8e85603fc300f0eec067dc6459b01b9d0e1005730979ba88"} Oct 30 21:56:07.079842 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:07.079827 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb1064fb8dcd42ab8e85603fc300f0eec067dc6459b01b9d0e1005730979ba88" Oct 30 21:56:08.743282 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:08.743249 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8709fb3-zffc6"] Oct 30 21:56:08.793583 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:08.792833 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8709fb3-zffc6"] Oct 30 21:56:10.421016 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:10.420982 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="243e1423-410b-4e99-88d5-59010335385a" path="/var/lib/kubelet/pods/243e1423-410b-4e99-88d5-59010335385a/volumes" Oct 30 21:56:10.494463 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:10.494419 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r"] Oct 30 21:56:10.494730 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:10.494706 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r" podUID="ca6f3e2f-0392-4d98-a47f-65deafaf93fb" containerName="istio-proxy" containerID="cri-o://45bc329e0c1c9088945e82cf7033fec15f9e933b72d3825d2201bc6805353a5b" gracePeriod=30 Oct 30 21:56:10.543024 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:10.542981 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9"] Oct 30 21:56:10.543224 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:10.543197 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9" podUID="26dc7eb4-1548-433d-b7dd-0368d36a466b" containerName="istio-proxy" containerID="cri-o://5b2ac266fc4a7d605d6f354af957294d12a393de1333396aef074274c65fded5" gracePeriod=30 Oct 30 21:56:11.387747 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:11.387701 5748 patch_prober.go:29] interesting pod/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:56:11.387934 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:11.387776 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9" podUID="26dc7eb4-1548-433d-b7dd-0368d36a466b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:56:13.121964 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:13.121931 5748 generic.go:356] "Generic (PLEG): container finished" podID="f3d0a7fa-e727-4776-b3c2-db1218760332" containerID="eaa55b38e0c967494780c31fd90097d9a1a29c32f3bfc7f22c3245291dbda6ca" exitCode=0 Oct 30 21:56:13.122419 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:13.121985 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh" event={"ID":"f3d0a7fa-e727-4776-b3c2-db1218760332","Type":"ContainerDied","Data":"eaa55b38e0c967494780c31fd90097d9a1a29c32f3bfc7f22c3245291dbda6ca"} Oct 30 21:56:13.122546 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:13.122518 5748 scope.go:122] "RemoveContainer" containerID="eaa55b38e0c967494780c31fd90097d9a1a29c32f3bfc7f22c3245291dbda6ca" Oct 30 21:56:13.124312 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:13.124281 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:56:13.124403 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:13.124351 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:56:13.124403 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:13.124384 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:56:14.133994 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:14.133957 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh" event={"ID":"f3d0a7fa-e727-4776-b3c2-db1218760332","Type":"ContainerStarted","Data":"dc83b0f53a3fff3f61080c7c335b5bd0ef3a17e911eb6fa0cbd4d10350728fc4"} Oct 30 21:56:14.569942 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:14.569880 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh" Oct 30 21:56:14.570155 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:14.570130 5748 patch_prober.go:29] interesting pod/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 30 21:56:14.570241 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:14.570186 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh" podUID="f3d0a7fa-e727-4776-b3c2-db1218760332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 30 21:56:14.945919 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:14.945874 5748 patch_prober.go:29] interesting pod/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:56:14.946098 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:14.945945 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r" podUID="ca6f3e2f-0392-4d98-a47f-65deafaf93fb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:56:15.574912 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:15.574862 5748 patch_prober.go:29] interesting pod/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 30 21:56:15.575292 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:15.574941 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh" podUID="f3d0a7fa-e727-4776-b3c2-db1218760332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 30 21:56:15.886746 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:15.886716 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r" Oct 30 21:56:15.969741 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:15.969719 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9" Oct 30 21:56:15.972149 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:15.972127 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-credential-socket\") pod \"ca6f3e2f-0392-4d98-a47f-65deafaf93fb\" (UID: \"ca6f3e2f-0392-4d98-a47f-65deafaf93fb\") " Oct 30 21:56:15.972265 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:15.972199 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-istio-envoy\") pod \"ca6f3e2f-0392-4d98-a47f-65deafaf93fb\" (UID: \"ca6f3e2f-0392-4d98-a47f-65deafaf93fb\") " Oct 30 21:56:15.972325 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:15.972285 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-kube-api-access-tflnp\" (UniqueName: \"kubernetes.io/projected/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-kube-api-access-tflnp\") pod \"ca6f3e2f-0392-4d98-a47f-65deafaf93fb\" (UID: \"ca6f3e2f-0392-4d98-a47f-65deafaf93fb\") " Oct 30 21:56:15.972381 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:15.972327 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-istio-podinfo\") pod \"ca6f3e2f-0392-4d98-a47f-65deafaf93fb\" (UID: \"ca6f3e2f-0392-4d98-a47f-65deafaf93fb\") " Oct 30 21:56:15.972434 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:15.972380 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-credential-socket" pod "ca6f3e2f-0392-4d98-a47f-65deafaf93fb" (UID: "ca6f3e2f-0392-4d98-a47f-65deafaf93fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:15.972434 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:15.972412 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-workload-socket\") pod \"ca6f3e2f-0392-4d98-a47f-65deafaf93fb\" (UID: \"ca6f3e2f-0392-4d98-a47f-65deafaf93fb\") " Oct 30 21:56:15.972528 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:15.972458 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-istio-token\" (UniqueName: \"kubernetes.io/projected/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-istio-token\") pod \"ca6f3e2f-0392-4d98-a47f-65deafaf93fb\" (UID: \"ca6f3e2f-0392-4d98-a47f-65deafaf93fb\") " Oct 30 21:56:15.972572 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:15.972552 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-istiod-ca-cert\") pod \"ca6f3e2f-0392-4d98-a47f-65deafaf93fb\" (UID: \"ca6f3e2f-0392-4d98-a47f-65deafaf93fb\") " Oct 30 21:56:15.972624 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:15.972615 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-istio-data\") pod \"ca6f3e2f-0392-4d98-a47f-65deafaf93fb\" (UID: \"ca6f3e2f-0392-4d98-a47f-65deafaf93fb\") " Oct 30 21:56:15.972673 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:15.972647 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-workload-certs\") pod \"ca6f3e2f-0392-4d98-a47f-65deafaf93fb\" (UID: \"ca6f3e2f-0392-4d98-a47f-65deafaf93fb\") " Oct 30 21:56:15.972890 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:15.972867 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-istiod-ca-cert" pod "ca6f3e2f-0392-4d98-a47f-65deafaf93fb" (UID: "ca6f3e2f-0392-4d98-a47f-65deafaf93fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:15.973087 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:15.973064 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-workload-socket" pod "ca6f3e2f-0392-4d98-a47f-65deafaf93fb" (UID: "ca6f3e2f-0392-4d98-a47f-65deafaf93fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:15.973283 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:15.973266 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-istio-data" pod "ca6f3e2f-0392-4d98-a47f-65deafaf93fb" (UID: "ca6f3e2f-0392-4d98-a47f-65deafaf93fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:15.973397 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:15.973300 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-workload-certs" pod "ca6f3e2f-0392-4d98-a47f-65deafaf93fb" (UID: "ca6f3e2f-0392-4d98-a47f-65deafaf93fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:15.973530 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:15.973510 5748 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-workload-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:15.973596 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:15.973536 5748 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-istiod-ca-cert\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:15.973596 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:15.973551 5748 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-istio-data\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:15.973596 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:15.973567 5748 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-workload-certs\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:15.973596 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:15.973582 5748 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-credential-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:15.974591 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:15.974566 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-istio-token" pod "ca6f3e2f-0392-4d98-a47f-65deafaf93fb" (UID: "ca6f3e2f-0392-4d98-a47f-65deafaf93fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:15.974734 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:15.974716 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-istio-envoy" pod "ca6f3e2f-0392-4d98-a47f-65deafaf93fb" (UID: "ca6f3e2f-0392-4d98-a47f-65deafaf93fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:15.975142 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:15.975121 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-kube-api-access-tflnp" pod "ca6f3e2f-0392-4d98-a47f-65deafaf93fb" (UID: "ca6f3e2f-0392-4d98-a47f-65deafaf93fb"). InnerVolumeSpecName "kube-api-access-tflnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:15.975608 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:15.975576 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-istio-podinfo" pod "ca6f3e2f-0392-4d98-a47f-65deafaf93fb" (UID: "ca6f3e2f-0392-4d98-a47f-65deafaf93fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:56:16.074012 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:16.073985 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26dc7eb4-1548-433d-b7dd-0368d36a466b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26dc7eb4-1548-433d-b7dd-0368d36a466b-credential-socket\") pod \"26dc7eb4-1548-433d-b7dd-0368d36a466b\" (UID: \"26dc7eb4-1548-433d-b7dd-0368d36a466b\") " Oct 30 21:56:16.074168 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:16.074039 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/26dc7eb4-1548-433d-b7dd-0368d36a466b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26dc7eb4-1548-433d-b7dd-0368d36a466b-istio-podinfo\") pod \"26dc7eb4-1548-433d-b7dd-0368d36a466b\" (UID: \"26dc7eb4-1548-433d-b7dd-0368d36a466b\") " Oct 30 21:56:16.074168 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:16.074141 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26dc7eb4-1548-433d-b7dd-0368d36a466b-istio-token\" (UniqueName: \"kubernetes.io/projected/26dc7eb4-1548-433d-b7dd-0368d36a466b-istio-token\") pod \"26dc7eb4-1548-433d-b7dd-0368d36a466b\" (UID: \"26dc7eb4-1548-433d-b7dd-0368d36a466b\") " Oct 30 21:56:16.074168 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:16.074164 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26dc7eb4-1548-433d-b7dd-0368d36a466b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26dc7eb4-1548-433d-b7dd-0368d36a466b-workload-certs\") pod \"26dc7eb4-1548-433d-b7dd-0368d36a466b\" (UID: \"26dc7eb4-1548-433d-b7dd-0368d36a466b\") " Oct 30 21:56:16.074330 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:16.074195 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26dc7eb4-1548-433d-b7dd-0368d36a466b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26dc7eb4-1548-433d-b7dd-0368d36a466b-istio-envoy\") pod \"26dc7eb4-1548-433d-b7dd-0368d36a466b\" (UID: \"26dc7eb4-1548-433d-b7dd-0368d36a466b\") " Oct 30 21:56:16.074330 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:16.074216 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26dc7eb4-1548-433d-b7dd-0368d36a466b-credential-socket" pod "26dc7eb4-1548-433d-b7dd-0368d36a466b" (UID: "26dc7eb4-1548-433d-b7dd-0368d36a466b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:16.074330 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:16.074289 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26dc7eb4-1548-433d-b7dd-0368d36a466b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26dc7eb4-1548-433d-b7dd-0368d36a466b-workload-socket\") pod \"26dc7eb4-1548-433d-b7dd-0368d36a466b\" (UID: \"26dc7eb4-1548-433d-b7dd-0368d36a466b\") " Oct 30 21:56:16.074491 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:16.074337 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26dc7eb4-1548-433d-b7dd-0368d36a466b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26dc7eb4-1548-433d-b7dd-0368d36a466b-istiod-ca-cert\") pod \"26dc7eb4-1548-433d-b7dd-0368d36a466b\" (UID: \"26dc7eb4-1548-433d-b7dd-0368d36a466b\") " Oct 30 21:56:16.074491 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:16.074366 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26dc7eb4-1548-433d-b7dd-0368d36a466b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26dc7eb4-1548-433d-b7dd-0368d36a466b-istio-data\") pod \"26dc7eb4-1548-433d-b7dd-0368d36a466b\" (UID: \"26dc7eb4-1548-433d-b7dd-0368d36a466b\") " Oct 30 21:56:16.074491 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:16.074444 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26dc7eb4-1548-433d-b7dd-0368d36a466b-workload-certs" pod "26dc7eb4-1548-433d-b7dd-0368d36a466b" (UID: "26dc7eb4-1548-433d-b7dd-0368d36a466b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:16.074641 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:16.074544 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26dc7eb4-1548-433d-b7dd-0368d36a466b-kube-api-access-qtr2f\" (UniqueName: \"kubernetes.io/projected/26dc7eb4-1548-433d-b7dd-0368d36a466b-kube-api-access-qtr2f\") pod \"26dc7eb4-1548-433d-b7dd-0368d36a466b\" (UID: \"26dc7eb4-1548-433d-b7dd-0368d36a466b\") " Oct 30 21:56:16.074641 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:16.074593 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26dc7eb4-1548-433d-b7dd-0368d36a466b-workload-socket" pod "26dc7eb4-1548-433d-b7dd-0368d36a466b" (UID: "26dc7eb4-1548-433d-b7dd-0368d36a466b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:16.074641 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:16.074601 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26dc7eb4-1548-433d-b7dd-0368d36a466b-istio-data" pod "26dc7eb4-1548-433d-b7dd-0368d36a466b" (UID: "26dc7eb4-1548-433d-b7dd-0368d36a466b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:16.075120 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:16.075095 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26dc7eb4-1548-433d-b7dd-0368d36a466b-istiod-ca-cert" pod "26dc7eb4-1548-433d-b7dd-0368d36a466b" (UID: "26dc7eb4-1548-433d-b7dd-0368d36a466b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:16.075255 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:16.075208 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tflnp\" (UniqueName: \"kubernetes.io/projected/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-kube-api-access-tflnp\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:16.075255 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:16.075221 5748 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-istio-podinfo\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:16.075255 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:16.075230 5748 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26dc7eb4-1548-433d-b7dd-0368d36a466b-workload-certs\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:16.075255 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:16.075238 5748 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-istio-token\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:16.075496 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:16.075263 5748 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26dc7eb4-1548-433d-b7dd-0368d36a466b-workload-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:16.075496 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:16.075277 5748 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26dc7eb4-1548-433d-b7dd-0368d36a466b-istiod-ca-cert\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:16.075496 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:16.075286 5748 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26dc7eb4-1548-433d-b7dd-0368d36a466b-istio-data\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:16.075496 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:16.075294 5748 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26dc7eb4-1548-433d-b7dd-0368d36a466b-credential-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:16.075496 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:16.075303 5748 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca6f3e2f-0392-4d98-a47f-65deafaf93fb-istio-envoy\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:16.076234 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:16.076211 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26dc7eb4-1548-433d-b7dd-0368d36a466b-istio-token" pod "26dc7eb4-1548-433d-b7dd-0368d36a466b" (UID: "26dc7eb4-1548-433d-b7dd-0368d36a466b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:16.076633 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:16.076613 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26dc7eb4-1548-433d-b7dd-0368d36a466b-istio-envoy" pod "26dc7eb4-1548-433d-b7dd-0368d36a466b" (UID: "26dc7eb4-1548-433d-b7dd-0368d36a466b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:16.076713 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:16.076631 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26dc7eb4-1548-433d-b7dd-0368d36a466b-istio-podinfo" pod "26dc7eb4-1548-433d-b7dd-0368d36a466b" (UID: "26dc7eb4-1548-433d-b7dd-0368d36a466b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:56:16.076713 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:16.076672 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26dc7eb4-1548-433d-b7dd-0368d36a466b-kube-api-access-qtr2f" pod "26dc7eb4-1548-433d-b7dd-0368d36a466b" (UID: "26dc7eb4-1548-433d-b7dd-0368d36a466b"). InnerVolumeSpecName "kube-api-access-qtr2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:16.151433 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:16.151335 5748 generic.go:356] "Generic (PLEG): container finished" podID="26dc7eb4-1548-433d-b7dd-0368d36a466b" containerID="5b2ac266fc4a7d605d6f354af957294d12a393de1333396aef074274c65fded5" exitCode=0 Oct 30 21:56:16.151433 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:16.151383 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9" Oct 30 21:56:16.152082 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:16.151377 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9" event={"ID":"26dc7eb4-1548-433d-b7dd-0368d36a466b","Type":"ContainerDied","Data":"5b2ac266fc4a7d605d6f354af957294d12a393de1333396aef074274c65fded5"} Oct 30 21:56:16.152082 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:16.151534 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9" event={"ID":"26dc7eb4-1548-433d-b7dd-0368d36a466b","Type":"ContainerDied","Data":"822ad153718a74a8cb7d365018ff7017e8c71c96191f306875f8615df507d05d"} Oct 30 21:56:16.152082 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:16.151568 5748 scope.go:122] "RemoveContainer" containerID="5b2ac266fc4a7d605d6f354af957294d12a393de1333396aef074274c65fded5" Oct 30 21:56:16.153662 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:16.153253 5748 generic.go:356] "Generic (PLEG): container finished" podID="ca6f3e2f-0392-4d98-a47f-65deafaf93fb" containerID="45bc329e0c1c9088945e82cf7033fec15f9e933b72d3825d2201bc6805353a5b" exitCode=0 Oct 30 21:56:16.153662 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:16.153303 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r" Oct 30 21:56:16.153662 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:16.153341 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r" event={"ID":"ca6f3e2f-0392-4d98-a47f-65deafaf93fb","Type":"ContainerDied","Data":"45bc329e0c1c9088945e82cf7033fec15f9e933b72d3825d2201bc6805353a5b"} Oct 30 21:56:16.153662 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:16.153381 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r" event={"ID":"ca6f3e2f-0392-4d98-a47f-65deafaf93fb","Type":"ContainerDied","Data":"662677d7f87d1d11e09c7d9286629ba71026a5a44afc0d3fd4731b84bdd5b644"} Oct 30 21:56:16.168581 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:16.168244 5748 scope.go:122] "RemoveContainer" containerID="5b2ac266fc4a7d605d6f354af957294d12a393de1333396aef074274c65fded5" Oct 30 21:56:16.168581 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:56:16.168559 5748 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b2ac266fc4a7d605d6f354af957294d12a393de1333396aef074274c65fded5\": container with ID starting with 5b2ac266fc4a7d605d6f354af957294d12a393de1333396aef074274c65fded5 not found: ID does not exist" containerID="5b2ac266fc4a7d605d6f354af957294d12a393de1333396aef074274c65fded5" Oct 30 21:56:16.168730 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:16.168599 5748 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b2ac266fc4a7d605d6f354af957294d12a393de1333396aef074274c65fded5"} err="failed to get container status \"5b2ac266fc4a7d605d6f354af957294d12a393de1333396aef074274c65fded5\": rpc error: code = NotFound desc = could not find container \"5b2ac266fc4a7d605d6f354af957294d12a393de1333396aef074274c65fded5\": container with ID starting with 5b2ac266fc4a7d605d6f354af957294d12a393de1333396aef074274c65fded5 not found: ID does not exist" Oct 30 21:56:16.168730 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:16.168624 5748 scope.go:122] "RemoveContainer" containerID="45bc329e0c1c9088945e82cf7033fec15f9e933b72d3825d2201bc6805353a5b" Oct 30 21:56:16.177335 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:16.177251 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtr2f\" (UniqueName: \"kubernetes.io/projected/26dc7eb4-1548-433d-b7dd-0368d36a466b-kube-api-access-qtr2f\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:16.177335 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:16.177285 5748 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26dc7eb4-1548-433d-b7dd-0368d36a466b-istio-podinfo\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:16.177335 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:16.177300 5748 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26dc7eb4-1548-433d-b7dd-0368d36a466b-istio-token\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:16.177335 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:16.177314 5748 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26dc7eb4-1548-433d-b7dd-0368d36a466b-istio-envoy\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:16.180920 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:16.180822 5748 scope.go:122] "RemoveContainer" containerID="45bc329e0c1c9088945e82cf7033fec15f9e933b72d3825d2201bc6805353a5b" Oct 30 21:56:16.181177 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:56:16.181142 5748 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45bc329e0c1c9088945e82cf7033fec15f9e933b72d3825d2201bc6805353a5b\": container with ID starting with 45bc329e0c1c9088945e82cf7033fec15f9e933b72d3825d2201bc6805353a5b not found: ID does not exist" containerID="45bc329e0c1c9088945e82cf7033fec15f9e933b72d3825d2201bc6805353a5b" Oct 30 21:56:16.181243 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:16.181179 5748 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45bc329e0c1c9088945e82cf7033fec15f9e933b72d3825d2201bc6805353a5b"} err="failed to get container status \"45bc329e0c1c9088945e82cf7033fec15f9e933b72d3825d2201bc6805353a5b\": rpc error: code = NotFound desc = could not find container \"45bc329e0c1c9088945e82cf7033fec15f9e933b72d3825d2201bc6805353a5b\": container with ID starting with 45bc329e0c1c9088945e82cf7033fec15f9e933b72d3825d2201bc6805353a5b not found: ID does not exist" Oct 30 21:56:16.411971 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:16.400664 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r"] Oct 30 21:56:16.432470 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:16.432443 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bmc5r"] Oct 30 21:56:16.577184 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:16.577147 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9"] Oct 30 21:56:16.579981 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:16.579523 5748 patch_prober.go:29] interesting pod/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 30 21:56:16.579981 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:16.579588 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh" podUID="f3d0a7fa-e727-4776-b3c2-db1218760332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 30 21:56:16.651663 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:16.651625 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-nnpj9"] Oct 30 21:56:17.570100 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:17.570065 5748 patch_prober.go:29] interesting pod/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 30 21:56:17.570310 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:17.570153 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh" podUID="f3d0a7fa-e727-4776-b3c2-db1218760332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 30 21:56:18.021039 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:18.021000 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-79e10a-55575bc788-b6qw2"] Oct 30 21:56:18.470919 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:18.468354 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26dc7eb4-1548-433d-b7dd-0368d36a466b" path="/var/lib/kubelet/pods/26dc7eb4-1548-433d-b7dd-0368d36a466b/volumes" Oct 30 21:56:18.470919 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:18.469430 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca6f3e2f-0392-4d98-a47f-65deafaf93fb" path="/var/lib/kubelet/pods/ca6f3e2f-0392-4d98-a47f-65deafaf93fb/volumes" Oct 30 21:56:18.569518 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:18.569484 5748 patch_prober.go:29] interesting pod/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 30 21:56:18.569713 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:18.569535 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh" podUID="f3d0a7fa-e727-4776-b3c2-db1218760332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 30 21:56:19.580917 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:19.580669 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh" Oct 30 21:56:19.588318 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:19.581707 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh" Oct 30 21:56:19.588318 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:19.583952 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh" Oct 30 21:56:23.022171 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:23.022104 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-79e10a-55575bc788-b6qw2" podUID="621fdae0-e3ad-43a6-a556-b7c55979f081" containerName="payload-processing" containerID="cri-o://5540fe39816fa508dbc08d8f263ebaf35d5770b25caca2eb327475d6d860aefa" gracePeriod=25 Oct 30 21:56:23.211616 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:23.211585 5748 generic.go:356] "Generic (PLEG): container finished" podID="621fdae0-e3ad-43a6-a556-b7c55979f081" containerID="5540fe39816fa508dbc08d8f263ebaf35d5770b25caca2eb327475d6d860aefa" exitCode=0 Oct 30 21:56:23.211800 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:23.211645 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-79e10a-55575bc788-b6qw2" event={"ID":"621fdae0-e3ad-43a6-a556-b7c55979f081","Type":"ContainerDied","Data":"5540fe39816fa508dbc08d8f263ebaf35d5770b25caca2eb327475d6d860aefa"} Oct 30 21:56:23.310012 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:23.309984 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-79e10a-55575bc788-b6qw2" Oct 30 21:56:23.411448 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:23.411411 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/621fdae0-e3ad-43a6-a556-b7c55979f081-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/621fdae0-e3ad-43a6-a556-b7c55979f081-plugins-config-volume\") pod \"621fdae0-e3ad-43a6-a556-b7c55979f081\" (UID: \"621fdae0-e3ad-43a6-a556-b7c55979f081\") " Oct 30 21:56:23.411621 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:23.411496 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/621fdae0-e3ad-43a6-a556-b7c55979f081-kube-api-access-8qkl8\" (UniqueName: \"kubernetes.io/projected/621fdae0-e3ad-43a6-a556-b7c55979f081-kube-api-access-8qkl8\") pod \"621fdae0-e3ad-43a6-a556-b7c55979f081\" (UID: \"621fdae0-e3ad-43a6-a556-b7c55979f081\") " Oct 30 21:56:23.411781 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:23.411755 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/621fdae0-e3ad-43a6-a556-b7c55979f081-plugins-config-volume" pod "621fdae0-e3ad-43a6-a556-b7c55979f081" (UID: "621fdae0-e3ad-43a6-a556-b7c55979f081"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:23.411950 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:23.411932 5748 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/621fdae0-e3ad-43a6-a556-b7c55979f081-plugins-config-volume\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:23.413582 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:23.413560 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/621fdae0-e3ad-43a6-a556-b7c55979f081-kube-api-access-8qkl8" pod "621fdae0-e3ad-43a6-a556-b7c55979f081" (UID: "621fdae0-e3ad-43a6-a556-b7c55979f081"). InnerVolumeSpecName "kube-api-access-8qkl8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:23.513300 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:23.513263 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qkl8\" (UniqueName: \"kubernetes.io/projected/621fdae0-e3ad-43a6-a556-b7c55979f081-kube-api-access-8qkl8\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:24.219477 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:24.219448 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-79e10a-55575bc788-b6qw2" Oct 30 21:56:24.219909 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:24.219483 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-79e10a-55575bc788-b6qw2" event={"ID":"621fdae0-e3ad-43a6-a556-b7c55979f081","Type":"ContainerDied","Data":"e9fcfb91b606bbd8d8f75e0d9e603236396dad7336ec56124ee43bfc467082be"} Oct 30 21:56:24.219909 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:24.219544 5748 scope.go:122] "RemoveContainer" containerID="5540fe39816fa508dbc08d8f263ebaf35d5770b25caca2eb327475d6d860aefa" Oct 30 21:56:24.395463 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:24.395433 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-79e10a-55575bc788-b6qw2"] Oct 30 21:56:24.445622 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:24.445591 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-79e10a-55575bc788-b6qw2"] Oct 30 21:56:26.418158 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:26.418124 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="621fdae0-e3ad-43a6-a556-b7c55979f081" path="/var/lib/kubelet/pods/621fdae0-e3ad-43a6-a556-b7c55979f081/volumes" Oct 30 21:56:30.145406 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:56:30.145366 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:56:36.798077 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:36.798044 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl"] Oct 30 21:56:36.798687 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:36.798672 5748 cpu_manager.go:414] "containerMap: removing container" podUID="621fdae0-e3ad-43a6-a556-b7c55979f081" containerName="payload-processing" Oct 30 21:56:36.798731 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:36.798689 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="621fdae0-e3ad-43a6-a556-b7c55979f081" containerName="payload-processing" Oct 30 21:56:36.798731 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:36.798709 5748 cpu_manager.go:414] "containerMap: removing container" podUID="26dc7eb4-1548-433d-b7dd-0368d36a466b" containerName="istio-proxy" Oct 30 21:56:36.798731 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:36.798714 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26dc7eb4-1548-433d-b7dd-0368d36a466b" containerName="istio-proxy" Oct 30 21:56:36.798824 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:36.798739 5748 cpu_manager.go:414] "containerMap: removing container" podUID="85e4ecda-eb68-4c06-abb0-b6b8ea5db24c" containerName="revoke-keys" Oct 30 21:56:36.798824 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:36.798744 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85e4ecda-eb68-4c06-abb0-b6b8ea5db24c" containerName="revoke-keys" Oct 30 21:56:36.798824 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:36.798753 5748 cpu_manager.go:414] "containerMap: removing container" podUID="ca6f3e2f-0392-4d98-a47f-65deafaf93fb" containerName="istio-proxy" Oct 30 21:56:36.798824 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:36.798757 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca6f3e2f-0392-4d98-a47f-65deafaf93fb" containerName="istio-proxy" Oct 30 21:56:36.799040 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:36.798851 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="621fdae0-e3ad-43a6-a556-b7c55979f081" containerName="payload-processing" Oct 30 21:56:36.799040 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:36.798862 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="26dc7eb4-1548-433d-b7dd-0368d36a466b" containerName="istio-proxy" Oct 30 21:56:36.799040 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:36.798873 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="85e4ecda-eb68-4c06-abb0-b6b8ea5db24c" containerName="revoke-keys" Oct 30 21:56:36.799040 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:36.798879 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca6f3e2f-0392-4d98-a47f-65deafaf93fb" containerName="istio-proxy" Oct 30 21:56:36.803590 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:36.803572 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl" Oct 30 21:56:36.856718 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:36.856672 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-5bfa09-openshift-default-dockercfg-jl9rm\"" Oct 30 21:56:36.888355 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:36.888325 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e231fcd9-6b46-4f08-bed5-947a82dc6d03-istio-envoy\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl\" (UID: \"e231fcd9-6b46-4f08-bed5-947a82dc6d03\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl" Oct 30 21:56:36.888604 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:36.888577 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj79s\" (UniqueName: \"kubernetes.io/projected/e231fcd9-6b46-4f08-bed5-947a82dc6d03-kube-api-access-tj79s\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl\" (UID: \"e231fcd9-6b46-4f08-bed5-947a82dc6d03\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl" Oct 30 21:56:36.888729 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:36.888707 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e231fcd9-6b46-4f08-bed5-947a82dc6d03-workload-socket\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl\" (UID: \"e231fcd9-6b46-4f08-bed5-947a82dc6d03\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl" Oct 30 21:56:36.888775 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:36.888746 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e231fcd9-6b46-4f08-bed5-947a82dc6d03-istio-data\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl\" (UID: \"e231fcd9-6b46-4f08-bed5-947a82dc6d03\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl" Oct 30 21:56:36.888819 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:36.888801 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e231fcd9-6b46-4f08-bed5-947a82dc6d03-istiod-ca-cert\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl\" (UID: \"e231fcd9-6b46-4f08-bed5-947a82dc6d03\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl" Oct 30 21:56:36.888856 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:36.888826 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e231fcd9-6b46-4f08-bed5-947a82dc6d03-istio-podinfo\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl\" (UID: \"e231fcd9-6b46-4f08-bed5-947a82dc6d03\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl" Oct 30 21:56:36.888914 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:36.888855 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e231fcd9-6b46-4f08-bed5-947a82dc6d03-credential-socket\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl\" (UID: \"e231fcd9-6b46-4f08-bed5-947a82dc6d03\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl" Oct 30 21:56:36.889081 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:36.889065 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e231fcd9-6b46-4f08-bed5-947a82dc6d03-istio-token\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl\" (UID: \"e231fcd9-6b46-4f08-bed5-947a82dc6d03\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl" Oct 30 21:56:36.889148 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:36.889099 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e231fcd9-6b46-4f08-bed5-947a82dc6d03-workload-certs\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl\" (UID: \"e231fcd9-6b46-4f08-bed5-947a82dc6d03\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl" Oct 30 21:56:36.989857 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:36.989821 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e231fcd9-6b46-4f08-bed5-947a82dc6d03-istiod-ca-cert\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl\" (UID: \"e231fcd9-6b46-4f08-bed5-947a82dc6d03\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl" Oct 30 21:56:36.989857 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:36.989862 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e231fcd9-6b46-4f08-bed5-947a82dc6d03-istio-podinfo\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl\" (UID: \"e231fcd9-6b46-4f08-bed5-947a82dc6d03\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl" Oct 30 21:56:36.990112 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:36.989888 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e231fcd9-6b46-4f08-bed5-947a82dc6d03-credential-socket\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl\" (UID: \"e231fcd9-6b46-4f08-bed5-947a82dc6d03\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl" Oct 30 21:56:36.990112 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:36.990005 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e231fcd9-6b46-4f08-bed5-947a82dc6d03-istio-token\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl\" (UID: \"e231fcd9-6b46-4f08-bed5-947a82dc6d03\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl" Oct 30 21:56:36.990112 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:36.990033 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e231fcd9-6b46-4f08-bed5-947a82dc6d03-workload-certs\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl\" (UID: \"e231fcd9-6b46-4f08-bed5-947a82dc6d03\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl" Oct 30 21:56:36.990112 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:36.990084 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e231fcd9-6b46-4f08-bed5-947a82dc6d03-istio-envoy\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl\" (UID: \"e231fcd9-6b46-4f08-bed5-947a82dc6d03\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl" Oct 30 21:56:36.990313 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:36.990214 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj79s\" (UniqueName: \"kubernetes.io/projected/e231fcd9-6b46-4f08-bed5-947a82dc6d03-kube-api-access-tj79s\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl\" (UID: \"e231fcd9-6b46-4f08-bed5-947a82dc6d03\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl" Oct 30 21:56:36.990370 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:36.990326 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e231fcd9-6b46-4f08-bed5-947a82dc6d03-workload-socket\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl\" (UID: \"e231fcd9-6b46-4f08-bed5-947a82dc6d03\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl" Oct 30 21:56:36.990370 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:36.990355 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e231fcd9-6b46-4f08-bed5-947a82dc6d03-istio-data\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl\" (UID: \"e231fcd9-6b46-4f08-bed5-947a82dc6d03\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl" Oct 30 21:56:36.990495 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:36.990385 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e231fcd9-6b46-4f08-bed5-947a82dc6d03-credential-socket\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl\" (UID: \"e231fcd9-6b46-4f08-bed5-947a82dc6d03\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl" Oct 30 21:56:36.990567 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:36.990497 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e231fcd9-6b46-4f08-bed5-947a82dc6d03-workload-certs\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl\" (UID: \"e231fcd9-6b46-4f08-bed5-947a82dc6d03\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl" Oct 30 21:56:36.990672 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:36.990643 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e231fcd9-6b46-4f08-bed5-947a82dc6d03-istiod-ca-cert\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl\" (UID: \"e231fcd9-6b46-4f08-bed5-947a82dc6d03\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl" Oct 30 21:56:36.990827 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:36.990690 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e231fcd9-6b46-4f08-bed5-947a82dc6d03-workload-socket\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl\" (UID: \"e231fcd9-6b46-4f08-bed5-947a82dc6d03\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl" Oct 30 21:56:36.990827 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:36.990759 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e231fcd9-6b46-4f08-bed5-947a82dc6d03-istio-data\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl\" (UID: \"e231fcd9-6b46-4f08-bed5-947a82dc6d03\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl" Oct 30 21:56:36.992305 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:36.992283 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e231fcd9-6b46-4f08-bed5-947a82dc6d03-istio-envoy\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl\" (UID: \"e231fcd9-6b46-4f08-bed5-947a82dc6d03\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl" Oct 30 21:56:36.992516 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:36.992499 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e231fcd9-6b46-4f08-bed5-947a82dc6d03-istio-podinfo\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl\" (UID: \"e231fcd9-6b46-4f08-bed5-947a82dc6d03\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl" Oct 30 21:56:37.006760 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:37.006734 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl"] Oct 30 21:56:37.052137 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:37.052063 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e231fcd9-6b46-4f08-bed5-947a82dc6d03-istio-token\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl\" (UID: \"e231fcd9-6b46-4f08-bed5-947a82dc6d03\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl" Oct 30 21:56:37.052275 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:37.052204 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj79s\" (UniqueName: \"kubernetes.io/projected/e231fcd9-6b46-4f08-bed5-947a82dc6d03-kube-api-access-tj79s\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl\" (UID: \"e231fcd9-6b46-4f08-bed5-947a82dc6d03\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl" Oct 30 21:56:37.116918 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:37.116871 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl" Oct 30 21:56:37.577531 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:37.577500 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl"] Oct 30 21:56:37.578596 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:56:37.578568 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode231fcd9_6b46_4f08_bed5_947a82dc6d03.slice/crio-bcf63c9b6caf1f34f68f4756735b77de3231ee408d302caa43b6115f22577559 WatchSource:0}: Error finding container bcf63c9b6caf1f34f68f4756735b77de3231ee408d302caa43b6115f22577559: Status 404 returned error can't find the container with id bcf63c9b6caf1f34f68f4756735b77de3231ee408d302caa43b6115f22577559 Oct 30 21:56:37.581292 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:37.581259 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:56:37.581385 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:37.581327 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:56:37.581385 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:37.581360 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:56:37.715528 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:37.712888 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-259e81-wbqvt"] Oct 30 21:56:37.732834 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:37.732794 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-259e81-wbqvt" Oct 30 21:56:37.776658 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:37.776626 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-259e81-wbqvt"] Oct 30 21:56:37.786664 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:37.786637 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:56:37.786876 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:37.786862 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:56:37.787111 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:37.787097 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:56:37.803289 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:37.803270 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 21:56:37.870580 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:37.870498 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57ae4154-bjptg"] Oct 30 21:56:37.875239 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:37.875218 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57ae4154-bjptg" Oct 30 21:56:37.903580 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:37.903534 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b5371f1-01b6-4b96-ace9-52a566fd635c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-259e81-wbqvt\" (UID: \"6b5371f1-01b6-4b96-ace9-52a566fd635c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-259e81-wbqvt" Oct 30 21:56:37.903741 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:37.903714 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b5371f1-01b6-4b96-ace9-52a566fd635c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-259e81-wbqvt\" (UID: \"6b5371f1-01b6-4b96-ace9-52a566fd635c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-259e81-wbqvt" Oct 30 21:56:37.903810 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:37.903750 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjfp5\" (UniqueName: \"kubernetes.io/projected/6b5371f1-01b6-4b96-ace9-52a566fd635c-kube-api-access-kjfp5\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-259e81-wbqvt\" (UID: \"6b5371f1-01b6-4b96-ace9-52a566fd635c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-259e81-wbqvt" Oct 30 21:56:37.934526 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:37.934499 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57ae4154-bjptg"] Oct 30 21:56:38.005657 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:38.005627 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/608b9e09-4acb-46eb-9ff9-f99d2fe04ce4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57ae4154-bjptg\" (UID: \"608b9e09-4acb-46eb-9ff9-f99d2fe04ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57ae4154-bjptg" Oct 30 21:56:38.005829 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:38.005750 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/608b9e09-4acb-46eb-9ff9-f99d2fe04ce4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57ae4154-bjptg\" (UID: \"608b9e09-4acb-46eb-9ff9-f99d2fe04ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57ae4154-bjptg" Oct 30 21:56:38.005829 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:38.005803 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b5371f1-01b6-4b96-ace9-52a566fd635c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-259e81-wbqvt\" (UID: \"6b5371f1-01b6-4b96-ace9-52a566fd635c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-259e81-wbqvt" Oct 30 21:56:38.006075 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:38.005834 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjfp5\" (UniqueName: \"kubernetes.io/projected/6b5371f1-01b6-4b96-ace9-52a566fd635c-kube-api-access-kjfp5\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-259e81-wbqvt\" (UID: \"6b5371f1-01b6-4b96-ace9-52a566fd635c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-259e81-wbqvt" Oct 30 21:56:38.006075 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:38.005971 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67pqh\" (UniqueName: \"kubernetes.io/projected/608b9e09-4acb-46eb-9ff9-f99d2fe04ce4-kube-api-access-67pqh\") pod \"maas-api-revoke-sub-57ae4154-bjptg\" (UID: \"608b9e09-4acb-46eb-9ff9-f99d2fe04ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57ae4154-bjptg" Oct 30 21:56:38.006196 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:38.006158 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b5371f1-01b6-4b96-ace9-52a566fd635c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-259e81-wbqvt\" (UID: \"6b5371f1-01b6-4b96-ace9-52a566fd635c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-259e81-wbqvt" Oct 30 21:56:38.006821 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:38.006801 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b5371f1-01b6-4b96-ace9-52a566fd635c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-259e81-wbqvt\" (UID: \"6b5371f1-01b6-4b96-ace9-52a566fd635c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-259e81-wbqvt" Oct 30 21:56:38.008380 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:38.008359 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b5371f1-01b6-4b96-ace9-52a566fd635c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-259e81-wbqvt\" (UID: \"6b5371f1-01b6-4b96-ace9-52a566fd635c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-259e81-wbqvt" Oct 30 21:56:38.067994 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:38.067962 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjfp5\" (UniqueName: \"kubernetes.io/projected/6b5371f1-01b6-4b96-ace9-52a566fd635c-kube-api-access-kjfp5\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-259e81-wbqvt\" (UID: \"6b5371f1-01b6-4b96-ace9-52a566fd635c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-259e81-wbqvt" Oct 30 21:56:38.107890 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:38.107847 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/608b9e09-4acb-46eb-9ff9-f99d2fe04ce4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57ae4154-bjptg\" (UID: \"608b9e09-4acb-46eb-9ff9-f99d2fe04ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57ae4154-bjptg" Oct 30 21:56:38.108097 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:38.107945 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/608b9e09-4acb-46eb-9ff9-f99d2fe04ce4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57ae4154-bjptg\" (UID: \"608b9e09-4acb-46eb-9ff9-f99d2fe04ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57ae4154-bjptg" Oct 30 21:56:38.108097 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:38.108051 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67pqh\" (UniqueName: \"kubernetes.io/projected/608b9e09-4acb-46eb-9ff9-f99d2fe04ce4-kube-api-access-67pqh\") pod \"maas-api-revoke-sub-57ae4154-bjptg\" (UID: \"608b9e09-4acb-46eb-9ff9-f99d2fe04ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57ae4154-bjptg" Oct 30 21:56:38.108869 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:38.108838 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/608b9e09-4acb-46eb-9ff9-f99d2fe04ce4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57ae4154-bjptg\" (UID: \"608b9e09-4acb-46eb-9ff9-f99d2fe04ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57ae4154-bjptg" Oct 30 21:56:38.110472 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:38.110443 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/608b9e09-4acb-46eb-9ff9-f99d2fe04ce4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57ae4154-bjptg\" (UID: \"608b9e09-4acb-46eb-9ff9-f99d2fe04ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57ae4154-bjptg" Oct 30 21:56:38.170368 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:38.170338 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67pqh\" (UniqueName: \"kubernetes.io/projected/608b9e09-4acb-46eb-9ff9-f99d2fe04ce4-kube-api-access-67pqh\") pod \"maas-api-revoke-sub-57ae4154-bjptg\" (UID: \"608b9e09-4acb-46eb-9ff9-f99d2fe04ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57ae4154-bjptg" Oct 30 21:56:38.190280 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:38.190252 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57ae4154-bjptg" Oct 30 21:56:38.334552 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:38.334505 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl" event={"ID":"e231fcd9-6b46-4f08-bed5-947a82dc6d03","Type":"ContainerStarted","Data":"0823e1f37d92dc72d7438c8313bf2fbb33675196767e4e28a0de8467e861c3a6"} Oct 30 21:56:38.334739 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:38.334578 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl" event={"ID":"e231fcd9-6b46-4f08-bed5-947a82dc6d03","Type":"ContainerStarted","Data":"bcf63c9b6caf1f34f68f4756735b77de3231ee408d302caa43b6115f22577559"} Oct 30 21:56:38.355786 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:38.355754 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-259e81-wbqvt" Oct 30 21:56:38.460932 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:38.460865 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl" podStartSLOduration=2.460851346 podStartE2EDuration="2.460851346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:38.457316723 +0000 UTC m=+1982.199837658" watchObservedRunningTime="2026-10-30 21:56:38.460851346 +0000 UTC m=+1982.203372281" Oct 30 21:56:38.599368 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:38.599332 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57ae4154-bjptg"] Oct 30 21:56:38.757279 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:38.757197 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-259e81-wbqvt"] Oct 30 21:56:38.762265 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:56:38.762237 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b5371f1_01b6_4b96_ace9_52a566fd635c.slice/crio-18b845dae852c49218c71a10c90ccd56554d292943d07a7d20029fe094264585 WatchSource:0}: Error finding container 18b845dae852c49218c71a10c90ccd56554d292943d07a7d20029fe094264585: Status 404 returned error can't find the container with id 18b845dae852c49218c71a10c90ccd56554d292943d07a7d20029fe094264585 Oct 30 21:56:39.122927 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:39.122207 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl" Oct 30 21:56:39.125064 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:39.125037 5748 patch_prober.go:29] interesting pod/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 21:56:39.125270 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:39.125239 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl" podUID="e231fcd9-6b46-4f08-bed5-947a82dc6d03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 21:56:39.343251 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:39.343171 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57ae4154-bjptg" event={"ID":"608b9e09-4acb-46eb-9ff9-f99d2fe04ce4","Type":"ContainerStarted","Data":"31c86636253bf4509a4063f7e5f34842af3902d12ab45bd08b64f61c4ee8c774"} Oct 30 21:56:39.343251 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:39.343221 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57ae4154-bjptg" event={"ID":"608b9e09-4acb-46eb-9ff9-f99d2fe04ce4","Type":"ContainerStarted","Data":"2a7dfbb643c21514121209f1111718e6149f063a1bbe1aed9cf6923c7069bfc3"} Oct 30 21:56:39.345008 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:39.344977 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-259e81-wbqvt" event={"ID":"6b5371f1-01b6-4b96-ace9-52a566fd635c","Type":"ContainerStarted","Data":"31a5ff5b1532ac417134169487fe3ce80338a2a2dff7291ecd625f3e958b0c57"} Oct 30 21:56:39.345119 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:39.345021 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-259e81-wbqvt" event={"ID":"6b5371f1-01b6-4b96-ace9-52a566fd635c","Type":"ContainerStarted","Data":"18b845dae852c49218c71a10c90ccd56554d292943d07a7d20029fe094264585"} Oct 30 21:56:39.464241 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:39.464190 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57ae4154-bjptg" podStartSLOduration=2.464175923 podStartE2EDuration="2.464175923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:39.460645824 +0000 UTC m=+1983.203166761" watchObservedRunningTime="2026-10-30 21:56:39.464175923 +0000 UTC m=+1983.206696857" Oct 30 21:56:39.588298 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:39.588250 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-259e81-wbqvt" podStartSLOduration=2.5882370420000003 podStartE2EDuration="2.588237042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:39.586880199 +0000 UTC m=+1983.329401139" watchObservedRunningTime="2026-10-30 21:56:39.588237042 +0000 UTC m=+1983.330757978" Oct 30 21:56:40.125333 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:40.125235 5748 patch_prober.go:29] interesting pod/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 21:56:40.125333 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:40.125289 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl" podUID="e231fcd9-6b46-4f08-bed5-947a82dc6d03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 21:56:40.352720 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:40.352686 5748 generic.go:356] "Generic (PLEG): container finished" podID="6b5371f1-01b6-4b96-ace9-52a566fd635c" containerID="31a5ff5b1532ac417134169487fe3ce80338a2a2dff7291ecd625f3e958b0c57" exitCode=0 Oct 30 21:56:40.352939 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:40.352773 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-259e81-wbqvt" event={"ID":"6b5371f1-01b6-4b96-ace9-52a566fd635c","Type":"ContainerDied","Data":"31a5ff5b1532ac417134169487fe3ce80338a2a2dff7291ecd625f3e958b0c57"} Oct 30 21:56:40.354447 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:40.354414 5748 generic.go:356] "Generic (PLEG): container finished" podID="608b9e09-4acb-46eb-9ff9-f99d2fe04ce4" containerID="31c86636253bf4509a4063f7e5f34842af3902d12ab45bd08b64f61c4ee8c774" exitCode=0 Oct 30 21:56:40.354581 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:40.354496 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57ae4154-bjptg" event={"ID":"608b9e09-4acb-46eb-9ff9-f99d2fe04ce4","Type":"ContainerDied","Data":"31c86636253bf4509a4063f7e5f34842af3902d12ab45bd08b64f61c4ee8c774"} Oct 30 21:56:41.119921 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:41.119029 5748 patch_prober.go:29] interesting pod/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 21:56:41.119921 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:41.119088 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl" podUID="e231fcd9-6b46-4f08-bed5-947a82dc6d03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 21:56:41.583208 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:41.583174 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57ae4154-bjptg" Oct 30 21:56:41.636258 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:41.636199 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-259e81-wbqvt" Oct 30 21:56:41.675183 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:41.675151 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b5371f1-01b6-4b96-ace9-52a566fd635c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b5371f1-01b6-4b96-ace9-52a566fd635c-maas-api-openshift-ca\") pod \"6b5371f1-01b6-4b96-ace9-52a566fd635c\" (UID: \"6b5371f1-01b6-4b96-ace9-52a566fd635c\") " Oct 30 21:56:41.675362 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:41.675198 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b5371f1-01b6-4b96-ace9-52a566fd635c-kube-api-access-kjfp5\" (UniqueName: \"kubernetes.io/projected/6b5371f1-01b6-4b96-ace9-52a566fd635c-kube-api-access-kjfp5\") pod \"6b5371f1-01b6-4b96-ace9-52a566fd635c\" (UID: \"6b5371f1-01b6-4b96-ace9-52a566fd635c\") " Oct 30 21:56:41.675362 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:41.675247 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/608b9e09-4acb-46eb-9ff9-f99d2fe04ce4-kube-api-access-67pqh\" (UniqueName: \"kubernetes.io/projected/608b9e09-4acb-46eb-9ff9-f99d2fe04ce4-kube-api-access-67pqh\") pod \"608b9e09-4acb-46eb-9ff9-f99d2fe04ce4\" (UID: \"608b9e09-4acb-46eb-9ff9-f99d2fe04ce4\") " Oct 30 21:56:41.675362 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:41.675268 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b5371f1-01b6-4b96-ace9-52a566fd635c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b5371f1-01b6-4b96-ace9-52a566fd635c-maas-api-xks-ca\") pod \"6b5371f1-01b6-4b96-ace9-52a566fd635c\" (UID: \"6b5371f1-01b6-4b96-ace9-52a566fd635c\") " Oct 30 21:56:41.675362 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:41.675316 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/608b9e09-4acb-46eb-9ff9-f99d2fe04ce4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/608b9e09-4acb-46eb-9ff9-f99d2fe04ce4-maas-api-xks-ca\") pod \"608b9e09-4acb-46eb-9ff9-f99d2fe04ce4\" (UID: \"608b9e09-4acb-46eb-9ff9-f99d2fe04ce4\") " Oct 30 21:56:41.675596 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:41.675482 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/608b9e09-4acb-46eb-9ff9-f99d2fe04ce4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/608b9e09-4acb-46eb-9ff9-f99d2fe04ce4-maas-api-openshift-ca\") pod \"608b9e09-4acb-46eb-9ff9-f99d2fe04ce4\" (UID: \"608b9e09-4acb-46eb-9ff9-f99d2fe04ce4\") " Oct 30 21:56:41.675596 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:41.675541 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b5371f1-01b6-4b96-ace9-52a566fd635c-maas-api-openshift-ca" pod "6b5371f1-01b6-4b96-ace9-52a566fd635c" (UID: "6b5371f1-01b6-4b96-ace9-52a566fd635c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:41.676211 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:41.676179 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/608b9e09-4acb-46eb-9ff9-f99d2fe04ce4-maas-api-openshift-ca" pod "608b9e09-4acb-46eb-9ff9-f99d2fe04ce4" (UID: "608b9e09-4acb-46eb-9ff9-f99d2fe04ce4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:41.676322 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:41.676269 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/608b9e09-4acb-46eb-9ff9-f99d2fe04ce4-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:41.676322 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:41.676291 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b5371f1-01b6-4b96-ace9-52a566fd635c-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:41.677658 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:41.677635 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b5371f1-01b6-4b96-ace9-52a566fd635c-maas-api-xks-ca" pod "6b5371f1-01b6-4b96-ace9-52a566fd635c" (UID: "6b5371f1-01b6-4b96-ace9-52a566fd635c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:56:41.677980 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:41.677965 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/608b9e09-4acb-46eb-9ff9-f99d2fe04ce4-kube-api-access-67pqh" pod "608b9e09-4acb-46eb-9ff9-f99d2fe04ce4" (UID: "608b9e09-4acb-46eb-9ff9-f99d2fe04ce4"). InnerVolumeSpecName "kube-api-access-67pqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:41.678099 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:41.678079 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/608b9e09-4acb-46eb-9ff9-f99d2fe04ce4-maas-api-xks-ca" pod "608b9e09-4acb-46eb-9ff9-f99d2fe04ce4" (UID: "608b9e09-4acb-46eb-9ff9-f99d2fe04ce4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:56:41.678099 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:41.678086 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b5371f1-01b6-4b96-ace9-52a566fd635c-kube-api-access-kjfp5" pod "6b5371f1-01b6-4b96-ace9-52a566fd635c" (UID: "6b5371f1-01b6-4b96-ace9-52a566fd635c"). InnerVolumeSpecName "kube-api-access-kjfp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:41.777294 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:41.777258 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjfp5\" (UniqueName: \"kubernetes.io/projected/6b5371f1-01b6-4b96-ace9-52a566fd635c-kube-api-access-kjfp5\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:41.777294 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:41.777292 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67pqh\" (UniqueName: \"kubernetes.io/projected/608b9e09-4acb-46eb-9ff9-f99d2fe04ce4-kube-api-access-67pqh\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:41.777294 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:41.777304 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b5371f1-01b6-4b96-ace9-52a566fd635c-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:41.777620 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:41.777315 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/608b9e09-4acb-46eb-9ff9-f99d2fe04ce4-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:42.124975 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:42.123023 5748 patch_prober.go:29] interesting pod/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 21:56:42.124975 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:42.123095 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl" podUID="e231fcd9-6b46-4f08-bed5-947a82dc6d03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 21:56:42.378169 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:42.378083 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-259e81-wbqvt" Oct 30 21:56:42.378169 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:42.378095 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-259e81-wbqvt" event={"ID":"6b5371f1-01b6-4b96-ace9-52a566fd635c","Type":"ContainerDied","Data":"18b845dae852c49218c71a10c90ccd56554d292943d07a7d20029fe094264585"} Oct 30 21:56:42.378169 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:42.378150 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18b845dae852c49218c71a10c90ccd56554d292943d07a7d20029fe094264585" Oct 30 21:56:42.379497 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:42.379474 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57ae4154-bjptg" Oct 30 21:56:42.379636 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:42.379508 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57ae4154-bjptg" event={"ID":"608b9e09-4acb-46eb-9ff9-f99d2fe04ce4","Type":"ContainerDied","Data":"2a7dfbb643c21514121209f1111718e6149f063a1bbe1aed9cf6923c7069bfc3"} Oct 30 21:56:42.379636 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:42.379550 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a7dfbb643c21514121209f1111718e6149f063a1bbe1aed9cf6923c7069bfc3" Oct 30 21:56:43.125164 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:43.125123 5748 patch_prober.go:29] interesting pod/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 21:56:43.125666 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:43.125196 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl" podUID="e231fcd9-6b46-4f08-bed5-947a82dc6d03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 21:56:44.122498 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:44.122462 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl" Oct 30 21:56:44.122950 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:44.122923 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl" Oct 30 21:56:44.123560 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:44.123541 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl" Oct 30 21:56:49.150347 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:49.150315 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2cf27b7-mkbc8"] Oct 30 21:56:49.150953 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:49.150936 5748 cpu_manager.go:414] "containerMap: removing container" podUID="608b9e09-4acb-46eb-9ff9-f99d2fe04ce4" containerName="revoke-keys" Oct 30 21:56:49.150953 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:49.150952 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="608b9e09-4acb-46eb-9ff9-f99d2fe04ce4" containerName="revoke-keys" Oct 30 21:56:49.151086 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:49.150970 5748 cpu_manager.go:414] "containerMap: removing container" podUID="6b5371f1-01b6-4b96-ace9-52a566fd635c" containerName="revoke-keys" Oct 30 21:56:49.151086 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:49.150976 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b5371f1-01b6-4b96-ace9-52a566fd635c" containerName="revoke-keys" Oct 30 21:56:49.151151 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:49.151087 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b5371f1-01b6-4b96-ace9-52a566fd635c" containerName="revoke-keys" Oct 30 21:56:49.151151 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:49.151105 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="608b9e09-4acb-46eb-9ff9-f99d2fe04ce4" containerName="revoke-keys" Oct 30 21:56:49.154906 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:49.154879 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2cf27b7-mkbc8" Oct 30 21:56:49.227452 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:49.227398 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:56:49.227452 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:49.227434 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:56:49.227697 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:49.227463 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:56:49.260593 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:49.260565 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 21:56:49.263505 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:49.263483 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2cf27b7-mkbc8"] Oct 30 21:56:49.279514 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:49.279479 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/946ae176-398f-4cdd-a2cd-fa71edfb6463-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2cf27b7-mkbc8\" (UID: \"946ae176-398f-4cdd-a2cd-fa71edfb6463\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2cf27b7-mkbc8" Oct 30 21:56:49.279686 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:49.279668 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5wd4\" (UniqueName: \"kubernetes.io/projected/946ae176-398f-4cdd-a2cd-fa71edfb6463-kube-api-access-z5wd4\") pod \"maas-api-revoke-sub-e2cf27b7-mkbc8\" (UID: \"946ae176-398f-4cdd-a2cd-fa71edfb6463\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2cf27b7-mkbc8" Oct 30 21:56:49.279911 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:49.279879 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/946ae176-398f-4cdd-a2cd-fa71edfb6463-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2cf27b7-mkbc8\" (UID: \"946ae176-398f-4cdd-a2cd-fa71edfb6463\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2cf27b7-mkbc8" Oct 30 21:56:49.381234 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:49.381210 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/946ae176-398f-4cdd-a2cd-fa71edfb6463-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2cf27b7-mkbc8\" (UID: \"946ae176-398f-4cdd-a2cd-fa71edfb6463\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2cf27b7-mkbc8" Oct 30 21:56:49.381386 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:49.381311 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/946ae176-398f-4cdd-a2cd-fa71edfb6463-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2cf27b7-mkbc8\" (UID: \"946ae176-398f-4cdd-a2cd-fa71edfb6463\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2cf27b7-mkbc8" Oct 30 21:56:49.381480 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:49.381463 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5wd4\" (UniqueName: \"kubernetes.io/projected/946ae176-398f-4cdd-a2cd-fa71edfb6463-kube-api-access-z5wd4\") pod \"maas-api-revoke-sub-e2cf27b7-mkbc8\" (UID: \"946ae176-398f-4cdd-a2cd-fa71edfb6463\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2cf27b7-mkbc8" Oct 30 21:56:49.381974 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:49.381954 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/946ae176-398f-4cdd-a2cd-fa71edfb6463-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2cf27b7-mkbc8\" (UID: \"946ae176-398f-4cdd-a2cd-fa71edfb6463\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2cf27b7-mkbc8" Oct 30 21:56:49.383713 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:49.383690 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/946ae176-398f-4cdd-a2cd-fa71edfb6463-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2cf27b7-mkbc8\" (UID: \"946ae176-398f-4cdd-a2cd-fa71edfb6463\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2cf27b7-mkbc8" Oct 30 21:56:49.443710 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:49.443637 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5wd4\" (UniqueName: \"kubernetes.io/projected/946ae176-398f-4cdd-a2cd-fa71edfb6463-kube-api-access-z5wd4\") pod \"maas-api-revoke-sub-e2cf27b7-mkbc8\" (UID: \"946ae176-398f-4cdd-a2cd-fa71edfb6463\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2cf27b7-mkbc8" Oct 30 21:56:49.472352 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:49.472324 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2cf27b7-mkbc8" Oct 30 21:56:49.855947 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:49.855920 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2cf27b7-mkbc8"] Oct 30 21:56:49.857086 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:56:49.857056 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod946ae176_398f_4cdd_a2cd_fa71edfb6463.slice/crio-56688779c122faaab8c72aecc48decb91b0fd5795ab07fcce38746a11425ce7d WatchSource:0}: Error finding container 56688779c122faaab8c72aecc48decb91b0fd5795ab07fcce38746a11425ce7d: Status 404 returned error can't find the container with id 56688779c122faaab8c72aecc48decb91b0fd5795ab07fcce38746a11425ce7d Oct 30 21:56:50.431703 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:50.431663 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2cf27b7-mkbc8" event={"ID":"946ae176-398f-4cdd-a2cd-fa71edfb6463","Type":"ContainerStarted","Data":"19d37c912a14439fbe5c98c0c4a522bceed8a728aeaf170ea2f0784e236d1e31"} Oct 30 21:56:50.432121 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:50.431726 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2cf27b7-mkbc8" event={"ID":"946ae176-398f-4cdd-a2cd-fa71edfb6463","Type":"ContainerStarted","Data":"56688779c122faaab8c72aecc48decb91b0fd5795ab07fcce38746a11425ce7d"} Oct 30 21:56:50.551928 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:50.551815 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2cf27b7-mkbc8" podStartSLOduration=1.551800444 podStartE2EDuration="1.551800444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:50.549561215 +0000 UTC m=+1994.292082150" watchObservedRunningTime="2026-10-30 21:56:50.551800444 +0000 UTC m=+1994.294321394" Oct 30 21:56:51.439574 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:51.439538 5748 generic.go:356] "Generic (PLEG): container finished" podID="946ae176-398f-4cdd-a2cd-fa71edfb6463" containerID="19d37c912a14439fbe5c98c0c4a522bceed8a728aeaf170ea2f0784e236d1e31" exitCode=0 Oct 30 21:56:51.440087 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:51.439625 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2cf27b7-mkbc8" event={"ID":"946ae176-398f-4cdd-a2cd-fa71edfb6463","Type":"ContainerDied","Data":"19d37c912a14439fbe5c98c0c4a522bceed8a728aeaf170ea2f0784e236d1e31"} Oct 30 21:56:52.435739 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:52.435710 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t"] Oct 30 21:56:52.435939 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:52.435914 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t" podUID="5e94f771-d6e2-4901-a055-04a4e905d8b8" containerName="istio-proxy" containerID="cri-o://cf907fad51e3e8cbf09b907bc79c3f4d3dd665bd3ef4c96744a153bb3c9d47c6" gracePeriod=30 Oct 30 21:56:52.610223 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:52.610197 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2cf27b7-mkbc8" Oct 30 21:56:52.724563 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:52.724490 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/946ae176-398f-4cdd-a2cd-fa71edfb6463-kube-api-access-z5wd4\" (UniqueName: \"kubernetes.io/projected/946ae176-398f-4cdd-a2cd-fa71edfb6463-kube-api-access-z5wd4\") pod \"946ae176-398f-4cdd-a2cd-fa71edfb6463\" (UID: \"946ae176-398f-4cdd-a2cd-fa71edfb6463\") " Oct 30 21:56:52.724563 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:52.724532 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/946ae176-398f-4cdd-a2cd-fa71edfb6463-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/946ae176-398f-4cdd-a2cd-fa71edfb6463-maas-api-xks-ca\") pod \"946ae176-398f-4cdd-a2cd-fa71edfb6463\" (UID: \"946ae176-398f-4cdd-a2cd-fa71edfb6463\") " Oct 30 21:56:52.724762 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:52.724665 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/946ae176-398f-4cdd-a2cd-fa71edfb6463-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/946ae176-398f-4cdd-a2cd-fa71edfb6463-maas-api-openshift-ca\") pod \"946ae176-398f-4cdd-a2cd-fa71edfb6463\" (UID: \"946ae176-398f-4cdd-a2cd-fa71edfb6463\") " Oct 30 21:56:52.725043 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:52.725016 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/946ae176-398f-4cdd-a2cd-fa71edfb6463-maas-api-openshift-ca" pod "946ae176-398f-4cdd-a2cd-fa71edfb6463" (UID: "946ae176-398f-4cdd-a2cd-fa71edfb6463"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:52.725246 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:52.725230 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/946ae176-398f-4cdd-a2cd-fa71edfb6463-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:52.726666 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:52.726641 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/946ae176-398f-4cdd-a2cd-fa71edfb6463-kube-api-access-z5wd4" pod "946ae176-398f-4cdd-a2cd-fa71edfb6463" (UID: "946ae176-398f-4cdd-a2cd-fa71edfb6463"). InnerVolumeSpecName "kube-api-access-z5wd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:52.726666 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:52.726643 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/946ae176-398f-4cdd-a2cd-fa71edfb6463-maas-api-xks-ca" pod "946ae176-398f-4cdd-a2cd-fa71edfb6463" (UID: "946ae176-398f-4cdd-a2cd-fa71edfb6463"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:56:52.826268 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:52.826238 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5wd4\" (UniqueName: \"kubernetes.io/projected/946ae176-398f-4cdd-a2cd-fa71edfb6463-kube-api-access-z5wd4\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:52.826268 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:52.826266 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/946ae176-398f-4cdd-a2cd-fa71edfb6463-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:53.461478 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:53.461431 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2cf27b7-mkbc8" event={"ID":"946ae176-398f-4cdd-a2cd-fa71edfb6463","Type":"ContainerDied","Data":"56688779c122faaab8c72aecc48decb91b0fd5795ab07fcce38746a11425ce7d"} Oct 30 21:56:53.461728 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:53.461708 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56688779c122faaab8c72aecc48decb91b0fd5795ab07fcce38746a11425ce7d" Oct 30 21:56:53.462028 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:53.462011 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2cf27b7-mkbc8" Oct 30 21:56:53.752295 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:53.752219 5748 patch_prober.go:29] interesting pod/e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:56:53.752295 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:53.752280 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t" podUID="5e94f771-d6e2-4901-a055-04a4e905d8b8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:56:56.190276 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:56.190239 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-khvpj"] Oct 30 21:56:56.190762 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:56.190432 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-khvpj" podUID="39e20f10-f92b-4919-b80d-519ad1120294" containerName="istio-proxy" containerID="cri-o://cf5f69ad3879667f2b7781df6e7f1bf46ba764ead9955350cac6e90f70cb4e15" gracePeriod=30 Oct 30 21:56:56.240820 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:56.240784 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-djvfd"] Oct 30 21:56:56.241027 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:56.241005 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-djvfd" podUID="325b45b9-2c8e-47a5-9559-7917be8270dd" containerName="istio-proxy" containerID="cri-o://6ad4c8c4c9b971250af3415b103221e3926864a02de566b3d8bc4f14f649fcfc" gracePeriod=30 Oct 30 21:56:57.711068 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:57.711013 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t" Oct 30 21:56:57.800696 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:57.800665 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e94f771-d6e2-4901-a055-04a4e905d8b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e94f771-d6e2-4901-a055-04a4e905d8b8-istio-data\") pod \"5e94f771-d6e2-4901-a055-04a4e905d8b8\" (UID: \"5e94f771-d6e2-4901-a055-04a4e905d8b8\") " Oct 30 21:56:57.800889 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:57.800707 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e94f771-d6e2-4901-a055-04a4e905d8b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e94f771-d6e2-4901-a055-04a4e905d8b8-workload-certs\") pod \"5e94f771-d6e2-4901-a055-04a4e905d8b8\" (UID: \"5e94f771-d6e2-4901-a055-04a4e905d8b8\") " Oct 30 21:56:57.800889 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:57.800730 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e94f771-d6e2-4901-a055-04a4e905d8b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e94f771-d6e2-4901-a055-04a4e905d8b8-istiod-ca-cert\") pod \"5e94f771-d6e2-4901-a055-04a4e905d8b8\" (UID: \"5e94f771-d6e2-4901-a055-04a4e905d8b8\") " Oct 30 21:56:57.800889 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:57.800754 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e94f771-d6e2-4901-a055-04a4e905d8b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e94f771-d6e2-4901-a055-04a4e905d8b8-istio-envoy\") pod \"5e94f771-d6e2-4901-a055-04a4e905d8b8\" (UID: \"5e94f771-d6e2-4901-a055-04a4e905d8b8\") " Oct 30 21:56:57.801134 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:57.800992 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e94f771-d6e2-4901-a055-04a4e905d8b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e94f771-d6e2-4901-a055-04a4e905d8b8-istio-podinfo\") pod \"5e94f771-d6e2-4901-a055-04a4e905d8b8\" (UID: \"5e94f771-d6e2-4901-a055-04a4e905d8b8\") " Oct 30 21:56:57.801134 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:57.800995 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e94f771-d6e2-4901-a055-04a4e905d8b8-workload-certs" pod "5e94f771-d6e2-4901-a055-04a4e905d8b8" (UID: "5e94f771-d6e2-4901-a055-04a4e905d8b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:57.801134 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:57.801030 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e94f771-d6e2-4901-a055-04a4e905d8b8-istio-token\" (UniqueName: \"kubernetes.io/projected/5e94f771-d6e2-4901-a055-04a4e905d8b8-istio-token\") pod \"5e94f771-d6e2-4901-a055-04a4e905d8b8\" (UID: \"5e94f771-d6e2-4901-a055-04a4e905d8b8\") " Oct 30 21:56:57.801134 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:57.801092 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e94f771-d6e2-4901-a055-04a4e905d8b8-kube-api-access-2bq9q\" (UniqueName: \"kubernetes.io/projected/5e94f771-d6e2-4901-a055-04a4e905d8b8-kube-api-access-2bq9q\") pod \"5e94f771-d6e2-4901-a055-04a4e905d8b8\" (UID: \"5e94f771-d6e2-4901-a055-04a4e905d8b8\") " Oct 30 21:56:57.801134 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:57.801012 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e94f771-d6e2-4901-a055-04a4e905d8b8-istio-data" pod "5e94f771-d6e2-4901-a055-04a4e905d8b8" (UID: "5e94f771-d6e2-4901-a055-04a4e905d8b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:57.801373 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:57.801159 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e94f771-d6e2-4901-a055-04a4e905d8b8-istiod-ca-cert" pod "5e94f771-d6e2-4901-a055-04a4e905d8b8" (UID: "5e94f771-d6e2-4901-a055-04a4e905d8b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:57.801373 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:57.801167 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e94f771-d6e2-4901-a055-04a4e905d8b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e94f771-d6e2-4901-a055-04a4e905d8b8-credential-socket\") pod \"5e94f771-d6e2-4901-a055-04a4e905d8b8\" (UID: \"5e94f771-d6e2-4901-a055-04a4e905d8b8\") " Oct 30 21:56:57.801373 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:57.801216 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e94f771-d6e2-4901-a055-04a4e905d8b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e94f771-d6e2-4901-a055-04a4e905d8b8-workload-socket\") pod \"5e94f771-d6e2-4901-a055-04a4e905d8b8\" (UID: \"5e94f771-d6e2-4901-a055-04a4e905d8b8\") " Oct 30 21:56:57.801516 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:57.801491 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e94f771-d6e2-4901-a055-04a4e905d8b8-workload-socket" pod "5e94f771-d6e2-4901-a055-04a4e905d8b8" (UID: "5e94f771-d6e2-4901-a055-04a4e905d8b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:57.801556 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:57.801527 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e94f771-d6e2-4901-a055-04a4e905d8b8-credential-socket" pod "5e94f771-d6e2-4901-a055-04a4e905d8b8" (UID: "5e94f771-d6e2-4901-a055-04a4e905d8b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:57.801849 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:57.801828 5748 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e94f771-d6e2-4901-a055-04a4e905d8b8-credential-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:57.801934 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:57.801854 5748 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e94f771-d6e2-4901-a055-04a4e905d8b8-workload-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:57.801934 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:57.801869 5748 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e94f771-d6e2-4901-a055-04a4e905d8b8-istio-data\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:57.801934 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:57.801884 5748 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e94f771-d6e2-4901-a055-04a4e905d8b8-workload-certs\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:57.801934 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:57.801915 5748 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e94f771-d6e2-4901-a055-04a4e905d8b8-istiod-ca-cert\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:57.803352 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:57.803322 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e94f771-d6e2-4901-a055-04a4e905d8b8-istio-podinfo" pod "5e94f771-d6e2-4901-a055-04a4e905d8b8" (UID: "5e94f771-d6e2-4901-a055-04a4e905d8b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:56:57.803461 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:57.803353 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e94f771-d6e2-4901-a055-04a4e905d8b8-istio-envoy" pod "5e94f771-d6e2-4901-a055-04a4e905d8b8" (UID: "5e94f771-d6e2-4901-a055-04a4e905d8b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:57.803645 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:57.803622 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e94f771-d6e2-4901-a055-04a4e905d8b8-kube-api-access-2bq9q" pod "5e94f771-d6e2-4901-a055-04a4e905d8b8" (UID: "5e94f771-d6e2-4901-a055-04a4e905d8b8"). InnerVolumeSpecName "kube-api-access-2bq9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:57.803721 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:57.803649 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e94f771-d6e2-4901-a055-04a4e905d8b8-istio-token" pod "5e94f771-d6e2-4901-a055-04a4e905d8b8" (UID: "5e94f771-d6e2-4901-a055-04a4e905d8b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:57.903981 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:57.903941 5748 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e94f771-d6e2-4901-a055-04a4e905d8b8-istio-podinfo\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:57.903981 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:57.903981 5748 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e94f771-d6e2-4901-a055-04a4e905d8b8-istio-token\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:57.904253 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:57.903997 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bq9q\" (UniqueName: \"kubernetes.io/projected/5e94f771-d6e2-4901-a055-04a4e905d8b8-kube-api-access-2bq9q\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:57.904253 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:57.904011 5748 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e94f771-d6e2-4901-a055-04a4e905d8b8-istio-envoy\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:58.496150 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:58.496118 5748 generic.go:356] "Generic (PLEG): container finished" podID="5e94f771-d6e2-4901-a055-04a4e905d8b8" containerID="cf907fad51e3e8cbf09b907bc79c3f4d3dd665bd3ef4c96744a153bb3c9d47c6" exitCode=0 Oct 30 21:56:58.496317 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:58.496177 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t" Oct 30 21:56:58.496317 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:58.496193 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t" event={"ID":"5e94f771-d6e2-4901-a055-04a4e905d8b8","Type":"ContainerDied","Data":"cf907fad51e3e8cbf09b907bc79c3f4d3dd665bd3ef4c96744a153bb3c9d47c6"} Oct 30 21:56:58.496317 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:58.496245 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t" event={"ID":"5e94f771-d6e2-4901-a055-04a4e905d8b8","Type":"ContainerDied","Data":"eca7bf3e66857343e72846f74d6d72cf62c3740e34fb6d7ac944eb2f684044b4"} Oct 30 21:56:58.496317 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:58.496267 5748 scope.go:122] "RemoveContainer" containerID="cf907fad51e3e8cbf09b907bc79c3f4d3dd665bd3ef4c96744a153bb3c9d47c6" Oct 30 21:56:58.506698 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:58.506679 5748 scope.go:122] "RemoveContainer" containerID="cf907fad51e3e8cbf09b907bc79c3f4d3dd665bd3ef4c96744a153bb3c9d47c6" Oct 30 21:56:58.506971 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:56:58.506949 5748 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf907fad51e3e8cbf09b907bc79c3f4d3dd665bd3ef4c96744a153bb3c9d47c6\": container with ID starting with cf907fad51e3e8cbf09b907bc79c3f4d3dd665bd3ef4c96744a153bb3c9d47c6 not found: ID does not exist" containerID="cf907fad51e3e8cbf09b907bc79c3f4d3dd665bd3ef4c96744a153bb3c9d47c6" Oct 30 21:56:58.507026 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:58.506978 5748 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf907fad51e3e8cbf09b907bc79c3f4d3dd665bd3ef4c96744a153bb3c9d47c6"} err="failed to get container status \"cf907fad51e3e8cbf09b907bc79c3f4d3dd665bd3ef4c96744a153bb3c9d47c6\": rpc error: code = NotFound desc = could not find container \"cf907fad51e3e8cbf09b907bc79c3f4d3dd665bd3ef4c96744a153bb3c9d47c6\": container with ID starting with cf907fad51e3e8cbf09b907bc79c3f4d3dd665bd3ef4c96744a153bb3c9d47c6 not found: ID does not exist" Oct 30 21:56:58.668017 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:58.667988 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t"] Oct 30 21:56:58.718787 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:58.718715 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-r8q7t"] Oct 30 21:56:59.139384 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:59.139347 5748 patch_prober.go:29] interesting pod/e2e-worker-w5-c7d477-openshift-default-776595695-khvpj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:56:59.139529 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:56:59.139420 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-khvpj" podUID="39e20f10-f92b-4919-b80d-519ad1120294" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:57:00.151591 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:57:00.151549 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:57:00.417823 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:00.417743 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e94f771-d6e2-4901-a055-04a4e905d8b8" path="/var/lib/kubelet/pods/5e94f771-d6e2-4901-a055-04a4e905d8b8/volumes" Oct 30 21:57:01.514372 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.514344 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-khvpj" Oct 30 21:57:01.518939 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.518879 5748 generic.go:356] "Generic (PLEG): container finished" podID="39e20f10-f92b-4919-b80d-519ad1120294" containerID="cf5f69ad3879667f2b7781df6e7f1bf46ba764ead9955350cac6e90f70cb4e15" exitCode=0 Oct 30 21:57:01.519092 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.518941 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-khvpj" Oct 30 21:57:01.519092 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.518983 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-khvpj" event={"ID":"39e20f10-f92b-4919-b80d-519ad1120294","Type":"ContainerDied","Data":"cf5f69ad3879667f2b7781df6e7f1bf46ba764ead9955350cac6e90f70cb4e15"} Oct 30 21:57:01.519092 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.519038 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-khvpj" event={"ID":"39e20f10-f92b-4919-b80d-519ad1120294","Type":"ContainerDied","Data":"7279a9eb1fa29f7e6a0e728ba20cb7e749c2565588b2815f115fdd571bef0741"} Oct 30 21:57:01.519092 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.519068 5748 scope.go:122] "RemoveContainer" containerID="cf5f69ad3879667f2b7781df6e7f1bf46ba764ead9955350cac6e90f70cb4e15" Oct 30 21:57:01.520791 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.520767 5748 generic.go:356] "Generic (PLEG): container finished" podID="325b45b9-2c8e-47a5-9559-7917be8270dd" containerID="6ad4c8c4c9b971250af3415b103221e3926864a02de566b3d8bc4f14f649fcfc" exitCode=0 Oct 30 21:57:01.520887 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.520820 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-djvfd" event={"ID":"325b45b9-2c8e-47a5-9559-7917be8270dd","Type":"ContainerDied","Data":"6ad4c8c4c9b971250af3415b103221e3926864a02de566b3d8bc4f14f649fcfc"} Oct 30 21:57:01.529827 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.529801 5748 scope.go:122] "RemoveContainer" containerID="cf5f69ad3879667f2b7781df6e7f1bf46ba764ead9955350cac6e90f70cb4e15" Oct 30 21:57:01.530144 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:57:01.530123 5748 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf5f69ad3879667f2b7781df6e7f1bf46ba764ead9955350cac6e90f70cb4e15\": container with ID starting with cf5f69ad3879667f2b7781df6e7f1bf46ba764ead9955350cac6e90f70cb4e15 not found: ID does not exist" containerID="cf5f69ad3879667f2b7781df6e7f1bf46ba764ead9955350cac6e90f70cb4e15" Oct 30 21:57:01.530218 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.530157 5748 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf5f69ad3879667f2b7781df6e7f1bf46ba764ead9955350cac6e90f70cb4e15"} err="failed to get container status \"cf5f69ad3879667f2b7781df6e7f1bf46ba764ead9955350cac6e90f70cb4e15\": rpc error: code = NotFound desc = could not find container \"cf5f69ad3879667f2b7781df6e7f1bf46ba764ead9955350cac6e90f70cb4e15\": container with ID starting with cf5f69ad3879667f2b7781df6e7f1bf46ba764ead9955350cac6e90f70cb4e15 not found: ID does not exist" Oct 30 21:57:01.551090 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.551066 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/39e20f10-f92b-4919-b80d-519ad1120294-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39e20f10-f92b-4919-b80d-519ad1120294-istio-podinfo\") pod \"39e20f10-f92b-4919-b80d-519ad1120294\" (UID: \"39e20f10-f92b-4919-b80d-519ad1120294\") " Oct 30 21:57:01.551189 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.551105 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39e20f10-f92b-4919-b80d-519ad1120294-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39e20f10-f92b-4919-b80d-519ad1120294-istio-envoy\") pod \"39e20f10-f92b-4919-b80d-519ad1120294\" (UID: \"39e20f10-f92b-4919-b80d-519ad1120294\") " Oct 30 21:57:01.551247 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.551188 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39e20f10-f92b-4919-b80d-519ad1120294-kube-api-access-jht5g\" (UniqueName: \"kubernetes.io/projected/39e20f10-f92b-4919-b80d-519ad1120294-kube-api-access-jht5g\") pod \"39e20f10-f92b-4919-b80d-519ad1120294\" (UID: \"39e20f10-f92b-4919-b80d-519ad1120294\") " Oct 30 21:57:01.551247 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.551215 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39e20f10-f92b-4919-b80d-519ad1120294-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39e20f10-f92b-4919-b80d-519ad1120294-workload-certs\") pod \"39e20f10-f92b-4919-b80d-519ad1120294\" (UID: \"39e20f10-f92b-4919-b80d-519ad1120294\") " Oct 30 21:57:01.551357 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.551248 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39e20f10-f92b-4919-b80d-519ad1120294-istio-token\" (UniqueName: \"kubernetes.io/projected/39e20f10-f92b-4919-b80d-519ad1120294-istio-token\") pod \"39e20f10-f92b-4919-b80d-519ad1120294\" (UID: \"39e20f10-f92b-4919-b80d-519ad1120294\") " Oct 30 21:57:01.551357 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.551275 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39e20f10-f92b-4919-b80d-519ad1120294-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39e20f10-f92b-4919-b80d-519ad1120294-workload-socket\") pod \"39e20f10-f92b-4919-b80d-519ad1120294\" (UID: \"39e20f10-f92b-4919-b80d-519ad1120294\") " Oct 30 21:57:01.551357 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.551330 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39e20f10-f92b-4919-b80d-519ad1120294-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39e20f10-f92b-4919-b80d-519ad1120294-istio-data\") pod \"39e20f10-f92b-4919-b80d-519ad1120294\" (UID: \"39e20f10-f92b-4919-b80d-519ad1120294\") " Oct 30 21:57:01.551517 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.551480 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39e20f10-f92b-4919-b80d-519ad1120294-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39e20f10-f92b-4919-b80d-519ad1120294-istiod-ca-cert\") pod \"39e20f10-f92b-4919-b80d-519ad1120294\" (UID: \"39e20f10-f92b-4919-b80d-519ad1120294\") " Oct 30 21:57:01.551572 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.551540 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39e20f10-f92b-4919-b80d-519ad1120294-workload-certs" pod "39e20f10-f92b-4919-b80d-519ad1120294" (UID: "39e20f10-f92b-4919-b80d-519ad1120294"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:01.551647 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.551573 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39e20f10-f92b-4919-b80d-519ad1120294-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39e20f10-f92b-4919-b80d-519ad1120294-credential-socket\") pod \"39e20f10-f92b-4919-b80d-519ad1120294\" (UID: \"39e20f10-f92b-4919-b80d-519ad1120294\") " Oct 30 21:57:01.551811 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.551788 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39e20f10-f92b-4919-b80d-519ad1120294-workload-socket" pod "39e20f10-f92b-4919-b80d-519ad1120294" (UID: "39e20f10-f92b-4919-b80d-519ad1120294"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:01.551878 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.551806 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39e20f10-f92b-4919-b80d-519ad1120294-istio-data" pod "39e20f10-f92b-4919-b80d-519ad1120294" (UID: "39e20f10-f92b-4919-b80d-519ad1120294"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:01.552037 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.552014 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39e20f10-f92b-4919-b80d-519ad1120294-credential-socket" pod "39e20f10-f92b-4919-b80d-519ad1120294" (UID: "39e20f10-f92b-4919-b80d-519ad1120294"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:01.552037 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.552022 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39e20f10-f92b-4919-b80d-519ad1120294-istiod-ca-cert" pod "39e20f10-f92b-4919-b80d-519ad1120294" (UID: "39e20f10-f92b-4919-b80d-519ad1120294"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:01.552339 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.552321 5748 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39e20f10-f92b-4919-b80d-519ad1120294-istiod-ca-cert\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:01.552386 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.552348 5748 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39e20f10-f92b-4919-b80d-519ad1120294-credential-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:01.552386 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.552362 5748 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39e20f10-f92b-4919-b80d-519ad1120294-workload-certs\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:01.552386 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.552376 5748 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39e20f10-f92b-4919-b80d-519ad1120294-workload-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:01.552494 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.552389 5748 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39e20f10-f92b-4919-b80d-519ad1120294-istio-data\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:01.553341 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.553321 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-djvfd" Oct 30 21:57:01.553552 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.553532 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39e20f10-f92b-4919-b80d-519ad1120294-kube-api-access-jht5g" pod "39e20f10-f92b-4919-b80d-519ad1120294" (UID: "39e20f10-f92b-4919-b80d-519ad1120294"). InnerVolumeSpecName "kube-api-access-jht5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:01.553632 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.553590 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39e20f10-f92b-4919-b80d-519ad1120294-istio-token" pod "39e20f10-f92b-4919-b80d-519ad1120294" (UID: "39e20f10-f92b-4919-b80d-519ad1120294"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:01.553831 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.553811 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39e20f10-f92b-4919-b80d-519ad1120294-istio-envoy" pod "39e20f10-f92b-4919-b80d-519ad1120294" (UID: "39e20f10-f92b-4919-b80d-519ad1120294"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:01.553875 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.553812 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39e20f10-f92b-4919-b80d-519ad1120294-istio-podinfo" pod "39e20f10-f92b-4919-b80d-519ad1120294" (UID: "39e20f10-f92b-4919-b80d-519ad1120294"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:57:01.653659 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.653601 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/325b45b9-2c8e-47a5-9559-7917be8270dd-istio-token\" (UniqueName: \"kubernetes.io/projected/325b45b9-2c8e-47a5-9559-7917be8270dd-istio-token\") pod \"325b45b9-2c8e-47a5-9559-7917be8270dd\" (UID: \"325b45b9-2c8e-47a5-9559-7917be8270dd\") " Oct 30 21:57:01.653907 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.653683 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/325b45b9-2c8e-47a5-9559-7917be8270dd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/325b45b9-2c8e-47a5-9559-7917be8270dd-istio-data\") pod \"325b45b9-2c8e-47a5-9559-7917be8270dd\" (UID: \"325b45b9-2c8e-47a5-9559-7917be8270dd\") " Oct 30 21:57:01.653907 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.653721 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/325b45b9-2c8e-47a5-9559-7917be8270dd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/325b45b9-2c8e-47a5-9559-7917be8270dd-istio-podinfo\") pod \"325b45b9-2c8e-47a5-9559-7917be8270dd\" (UID: \"325b45b9-2c8e-47a5-9559-7917be8270dd\") " Oct 30 21:57:01.653907 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.653795 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/325b45b9-2c8e-47a5-9559-7917be8270dd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/325b45b9-2c8e-47a5-9559-7917be8270dd-workload-socket\") pod \"325b45b9-2c8e-47a5-9559-7917be8270dd\" (UID: \"325b45b9-2c8e-47a5-9559-7917be8270dd\") " Oct 30 21:57:01.653907 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.653847 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/325b45b9-2c8e-47a5-9559-7917be8270dd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/325b45b9-2c8e-47a5-9559-7917be8270dd-workload-certs\") pod \"325b45b9-2c8e-47a5-9559-7917be8270dd\" (UID: \"325b45b9-2c8e-47a5-9559-7917be8270dd\") " Oct 30 21:57:01.654144 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.653918 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/325b45b9-2c8e-47a5-9559-7917be8270dd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/325b45b9-2c8e-47a5-9559-7917be8270dd-credential-socket\") pod \"325b45b9-2c8e-47a5-9559-7917be8270dd\" (UID: \"325b45b9-2c8e-47a5-9559-7917be8270dd\") " Oct 30 21:57:01.654144 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.653989 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/325b45b9-2c8e-47a5-9559-7917be8270dd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/325b45b9-2c8e-47a5-9559-7917be8270dd-istiod-ca-cert\") pod \"325b45b9-2c8e-47a5-9559-7917be8270dd\" (UID: \"325b45b9-2c8e-47a5-9559-7917be8270dd\") " Oct 30 21:57:01.654144 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.654034 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/325b45b9-2c8e-47a5-9559-7917be8270dd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/325b45b9-2c8e-47a5-9559-7917be8270dd-istio-envoy\") pod \"325b45b9-2c8e-47a5-9559-7917be8270dd\" (UID: \"325b45b9-2c8e-47a5-9559-7917be8270dd\") " Oct 30 21:57:01.654300 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.654233 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/325b45b9-2c8e-47a5-9559-7917be8270dd-kube-api-access-gcfgb\" (UniqueName: \"kubernetes.io/projected/325b45b9-2c8e-47a5-9559-7917be8270dd-kube-api-access-gcfgb\") pod \"325b45b9-2c8e-47a5-9559-7917be8270dd\" (UID: \"325b45b9-2c8e-47a5-9559-7917be8270dd\") " Oct 30 21:57:01.654300 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.654270 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/325b45b9-2c8e-47a5-9559-7917be8270dd-workload-socket" pod "325b45b9-2c8e-47a5-9559-7917be8270dd" (UID: "325b45b9-2c8e-47a5-9559-7917be8270dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:01.654952 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.654662 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/325b45b9-2c8e-47a5-9559-7917be8270dd-istio-data" pod "325b45b9-2c8e-47a5-9559-7917be8270dd" (UID: "325b45b9-2c8e-47a5-9559-7917be8270dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:01.654952 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.654684 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/325b45b9-2c8e-47a5-9559-7917be8270dd-credential-socket" pod "325b45b9-2c8e-47a5-9559-7917be8270dd" (UID: "325b45b9-2c8e-47a5-9559-7917be8270dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:01.654952 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.654742 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/325b45b9-2c8e-47a5-9559-7917be8270dd-istiod-ca-cert" pod "325b45b9-2c8e-47a5-9559-7917be8270dd" (UID: "325b45b9-2c8e-47a5-9559-7917be8270dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:01.654952 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.654859 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jht5g\" (UniqueName: \"kubernetes.io/projected/39e20f10-f92b-4919-b80d-519ad1120294-kube-api-access-jht5g\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:01.654952 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.654880 5748 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39e20f10-f92b-4919-b80d-519ad1120294-istio-token\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:01.654952 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.654931 5748 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/325b45b9-2c8e-47a5-9559-7917be8270dd-istio-data\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:01.654952 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.654947 5748 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/325b45b9-2c8e-47a5-9559-7917be8270dd-workload-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:01.654952 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.654961 5748 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/325b45b9-2c8e-47a5-9559-7917be8270dd-credential-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:01.655464 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.654975 5748 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39e20f10-f92b-4919-b80d-519ad1120294-istio-podinfo\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:01.655464 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.654989 5748 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39e20f10-f92b-4919-b80d-519ad1120294-istio-envoy\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:01.655464 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.655003 5748 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/325b45b9-2c8e-47a5-9559-7917be8270dd-istiod-ca-cert\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:01.655464 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.654927 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/325b45b9-2c8e-47a5-9559-7917be8270dd-workload-certs" pod "325b45b9-2c8e-47a5-9559-7917be8270dd" (UID: "325b45b9-2c8e-47a5-9559-7917be8270dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:01.657235 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.657211 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/325b45b9-2c8e-47a5-9559-7917be8270dd-istio-envoy" pod "325b45b9-2c8e-47a5-9559-7917be8270dd" (UID: "325b45b9-2c8e-47a5-9559-7917be8270dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:01.657235 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.657217 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/325b45b9-2c8e-47a5-9559-7917be8270dd-kube-api-access-gcfgb" pod "325b45b9-2c8e-47a5-9559-7917be8270dd" (UID: "325b45b9-2c8e-47a5-9559-7917be8270dd"). InnerVolumeSpecName "kube-api-access-gcfgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:01.657416 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.657290 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/325b45b9-2c8e-47a5-9559-7917be8270dd-istio-podinfo" pod "325b45b9-2c8e-47a5-9559-7917be8270dd" (UID: "325b45b9-2c8e-47a5-9559-7917be8270dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:57:01.657705 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.657685 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/325b45b9-2c8e-47a5-9559-7917be8270dd-istio-token" pod "325b45b9-2c8e-47a5-9559-7917be8270dd" (UID: "325b45b9-2c8e-47a5-9559-7917be8270dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:01.756643 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.756572 5748 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/325b45b9-2c8e-47a5-9559-7917be8270dd-workload-certs\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:01.756643 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.756597 5748 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/325b45b9-2c8e-47a5-9559-7917be8270dd-istio-envoy\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:01.756643 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.756615 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcfgb\" (UniqueName: \"kubernetes.io/projected/325b45b9-2c8e-47a5-9559-7917be8270dd-kube-api-access-gcfgb\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:01.756643 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.756624 5748 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/325b45b9-2c8e-47a5-9559-7917be8270dd-istio-token\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:01.756643 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.756634 5748 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/325b45b9-2c8e-47a5-9559-7917be8270dd-istio-podinfo\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:01.981856 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.981821 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-khvpj"] Oct 30 21:57:01.988241 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:01.988212 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-khvpj"] Oct 30 21:57:02.418429 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:02.418395 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39e20f10-f92b-4919-b80d-519ad1120294" path="/var/lib/kubelet/pods/39e20f10-f92b-4919-b80d-519ad1120294/volumes" Oct 30 21:57:02.530209 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:02.530120 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-djvfd" event={"ID":"325b45b9-2c8e-47a5-9559-7917be8270dd","Type":"ContainerDied","Data":"080dcb1860402d0161d70773d9f188bdc5befc4d959eefe444d4bd7ea085ad8c"} Oct 30 21:57:02.530209 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:02.530143 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-djvfd" Oct 30 21:57:02.530209 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:02.530189 5748 scope.go:122] "RemoveContainer" containerID="6ad4c8c4c9b971250af3415b103221e3926864a02de566b3d8bc4f14f649fcfc" Oct 30 21:57:02.699075 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:02.699041 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-djvfd"] Oct 30 21:57:02.701792 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:02.701765 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-djvfd"] Oct 30 21:57:04.417887 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:04.417851 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="325b45b9-2c8e-47a5-9559-7917be8270dd" path="/var/lib/kubelet/pods/325b45b9-2c8e-47a5-9559-7917be8270dd/volumes" Oct 30 21:57:07.450870 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:07.450827 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bxj6r"] Oct 30 21:57:07.451314 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:07.451033 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bxj6r" podUID="6c04ccfd-188d-4566-bf40-2732890754d6" containerName="istio-proxy" containerID="cri-o://4dde1979ea38af11d2d62f454c53385cee62f20bcad15fb3c9a84b69fd4727cf" gracePeriod=30 Oct 30 21:57:10.024535 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:10.024498 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e51c52-l8pv8"] Oct 30 21:57:10.025081 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:10.025066 5748 cpu_manager.go:414] "containerMap: removing container" podUID="946ae176-398f-4cdd-a2cd-fa71edfb6463" containerName="revoke-keys" Oct 30 21:57:10.025081 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:10.025081 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="946ae176-398f-4cdd-a2cd-fa71edfb6463" containerName="revoke-keys" Oct 30 21:57:10.025208 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:10.025130 5748 cpu_manager.go:414] "containerMap: removing container" podUID="39e20f10-f92b-4919-b80d-519ad1120294" containerName="istio-proxy" Oct 30 21:57:10.025208 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:10.025136 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39e20f10-f92b-4919-b80d-519ad1120294" containerName="istio-proxy" Oct 30 21:57:10.025208 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:10.025146 5748 cpu_manager.go:414] "containerMap: removing container" podUID="5e94f771-d6e2-4901-a055-04a4e905d8b8" containerName="istio-proxy" Oct 30 21:57:10.025208 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:10.025153 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e94f771-d6e2-4901-a055-04a4e905d8b8" containerName="istio-proxy" Oct 30 21:57:10.025208 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:10.025161 5748 cpu_manager.go:414] "containerMap: removing container" podUID="325b45b9-2c8e-47a5-9559-7917be8270dd" containerName="istio-proxy" Oct 30 21:57:10.025208 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:10.025166 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="325b45b9-2c8e-47a5-9559-7917be8270dd" containerName="istio-proxy" Oct 30 21:57:10.025382 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:10.025250 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="946ae176-398f-4cdd-a2cd-fa71edfb6463" containerName="revoke-keys" Oct 30 21:57:10.025382 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:10.025264 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="39e20f10-f92b-4919-b80d-519ad1120294" containerName="istio-proxy" Oct 30 21:57:10.025382 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:10.025270 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e94f771-d6e2-4901-a055-04a4e905d8b8" containerName="istio-proxy" Oct 30 21:57:10.025382 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:10.025275 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="325b45b9-2c8e-47a5-9559-7917be8270dd" containerName="istio-proxy" Oct 30 21:57:10.029955 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:10.029938 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e51c52-l8pv8" Oct 30 21:57:10.083597 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:10.083550 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:57:10.083597 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:10.083552 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:57:10.133844 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:10.133816 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:57:10.134021 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:10.133945 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 21:57:10.136723 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:10.136703 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e51c52-l8pv8"] Oct 30 21:57:10.159250 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:10.159224 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5da50f53-661c-4e80-b55b-4ae2ab155ceb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e51c52-l8pv8\" (UID: \"5da50f53-661c-4e80-b55b-4ae2ab155ceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e51c52-l8pv8" Oct 30 21:57:10.159385 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:10.159272 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5da50f53-661c-4e80-b55b-4ae2ab155ceb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e51c52-l8pv8\" (UID: \"5da50f53-661c-4e80-b55b-4ae2ab155ceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e51c52-l8pv8" Oct 30 21:57:10.159385 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:10.159359 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgt66\" (UniqueName: \"kubernetes.io/projected/5da50f53-661c-4e80-b55b-4ae2ab155ceb-kube-api-access-wgt66\") pod \"maas-api-revoke-keys-e2e-api-a-e51c52-l8pv8\" (UID: \"5da50f53-661c-4e80-b55b-4ae2ab155ceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e51c52-l8pv8" Oct 30 21:57:10.260253 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:10.260213 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgt66\" (UniqueName: \"kubernetes.io/projected/5da50f53-661c-4e80-b55b-4ae2ab155ceb-kube-api-access-wgt66\") pod \"maas-api-revoke-keys-e2e-api-a-e51c52-l8pv8\" (UID: \"5da50f53-661c-4e80-b55b-4ae2ab155ceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e51c52-l8pv8" Oct 30 21:57:10.260474 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:10.260455 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5da50f53-661c-4e80-b55b-4ae2ab155ceb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e51c52-l8pv8\" (UID: \"5da50f53-661c-4e80-b55b-4ae2ab155ceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e51c52-l8pv8" Oct 30 21:57:10.260561 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:10.260538 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5da50f53-661c-4e80-b55b-4ae2ab155ceb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e51c52-l8pv8\" (UID: \"5da50f53-661c-4e80-b55b-4ae2ab155ceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e51c52-l8pv8" Oct 30 21:57:10.261132 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:10.261108 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5da50f53-661c-4e80-b55b-4ae2ab155ceb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e51c52-l8pv8\" (UID: \"5da50f53-661c-4e80-b55b-4ae2ab155ceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e51c52-l8pv8" Oct 30 21:57:10.262883 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:10.262859 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5da50f53-661c-4e80-b55b-4ae2ab155ceb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e51c52-l8pv8\" (UID: \"5da50f53-661c-4e80-b55b-4ae2ab155ceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e51c52-l8pv8" Oct 30 21:57:10.321741 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:10.321685 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgt66\" (UniqueName: \"kubernetes.io/projected/5da50f53-661c-4e80-b55b-4ae2ab155ceb-kube-api-access-wgt66\") pod \"maas-api-revoke-keys-e2e-api-a-e51c52-l8pv8\" (UID: \"5da50f53-661c-4e80-b55b-4ae2ab155ceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e51c52-l8pv8" Oct 30 21:57:10.345654 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:10.345623 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e51c52-l8pv8" Oct 30 21:57:10.727138 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:10.727105 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e51c52-l8pv8"] Oct 30 21:57:10.728010 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:57:10.727977 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5da50f53_661c_4e80_b55b_4ae2ab155ceb.slice/crio-e176ff14190669edd60cee22b7296e50a57fd8db0de4960cc3439f71f3483675 WatchSource:0}: Error finding container e176ff14190669edd60cee22b7296e50a57fd8db0de4960cc3439f71f3483675: Status 404 returned error can't find the container with id e176ff14190669edd60cee22b7296e50a57fd8db0de4960cc3439f71f3483675 Oct 30 21:57:11.590491 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:11.590457 5748 generic.go:356] "Generic (PLEG): container finished" podID="5da50f53-661c-4e80-b55b-4ae2ab155ceb" containerID="2d3e4c676bc3a45af48d84132f8b4bcb45c7d54e18afd036a7a38f291aa92f7b" exitCode=0 Oct 30 21:57:11.590869 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:11.590532 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e51c52-l8pv8" event={"ID":"5da50f53-661c-4e80-b55b-4ae2ab155ceb","Type":"ContainerDied","Data":"2d3e4c676bc3a45af48d84132f8b4bcb45c7d54e18afd036a7a38f291aa92f7b"} Oct 30 21:57:11.590869 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:11.590598 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e51c52-l8pv8" event={"ID":"5da50f53-661c-4e80-b55b-4ae2ab155ceb","Type":"ContainerStarted","Data":"e176ff14190669edd60cee22b7296e50a57fd8db0de4960cc3439f71f3483675"} Oct 30 21:57:12.599290 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:12.599260 5748 generic.go:356] "Generic (PLEG): container finished" podID="6c04ccfd-188d-4566-bf40-2732890754d6" containerID="4dde1979ea38af11d2d62f454c53385cee62f20bcad15fb3c9a84b69fd4727cf" exitCode=0 Oct 30 21:57:12.599664 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:12.599332 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bxj6r" event={"ID":"6c04ccfd-188d-4566-bf40-2732890754d6","Type":"ContainerDied","Data":"4dde1979ea38af11d2d62f454c53385cee62f20bcad15fb3c9a84b69fd4727cf"} Oct 30 21:57:12.720546 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:12.720482 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bxj6r" Oct 30 21:57:12.788026 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:12.788003 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e51c52-l8pv8" Oct 30 21:57:12.794186 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:12.794162 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c04ccfd-188d-4566-bf40-2732890754d6-istio-token\" (UniqueName: \"kubernetes.io/projected/6c04ccfd-188d-4566-bf40-2732890754d6-istio-token\") pod \"6c04ccfd-188d-4566-bf40-2732890754d6\" (UID: \"6c04ccfd-188d-4566-bf40-2732890754d6\") " Oct 30 21:57:12.794295 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:12.794264 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c04ccfd-188d-4566-bf40-2732890754d6-kube-api-access-kzs59\" (UniqueName: \"kubernetes.io/projected/6c04ccfd-188d-4566-bf40-2732890754d6-kube-api-access-kzs59\") pod \"6c04ccfd-188d-4566-bf40-2732890754d6\" (UID: \"6c04ccfd-188d-4566-bf40-2732890754d6\") " Oct 30 21:57:12.794295 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:12.794289 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c04ccfd-188d-4566-bf40-2732890754d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c04ccfd-188d-4566-bf40-2732890754d6-credential-socket\") pod \"6c04ccfd-188d-4566-bf40-2732890754d6\" (UID: \"6c04ccfd-188d-4566-bf40-2732890754d6\") " Oct 30 21:57:12.794403 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:12.794319 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c04ccfd-188d-4566-bf40-2732890754d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c04ccfd-188d-4566-bf40-2732890754d6-istiod-ca-cert\") pod \"6c04ccfd-188d-4566-bf40-2732890754d6\" (UID: \"6c04ccfd-188d-4566-bf40-2732890754d6\") " Oct 30 21:57:12.794403 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:12.794372 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c04ccfd-188d-4566-bf40-2732890754d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c04ccfd-188d-4566-bf40-2732890754d6-workload-certs\") pod \"6c04ccfd-188d-4566-bf40-2732890754d6\" (UID: \"6c04ccfd-188d-4566-bf40-2732890754d6\") " Oct 30 21:57:12.794516 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:12.794411 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c04ccfd-188d-4566-bf40-2732890754d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c04ccfd-188d-4566-bf40-2732890754d6-istio-data\") pod \"6c04ccfd-188d-4566-bf40-2732890754d6\" (UID: \"6c04ccfd-188d-4566-bf40-2732890754d6\") " Oct 30 21:57:12.794516 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:12.794461 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c04ccfd-188d-4566-bf40-2732890754d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c04ccfd-188d-4566-bf40-2732890754d6-istio-envoy\") pod \"6c04ccfd-188d-4566-bf40-2732890754d6\" (UID: \"6c04ccfd-188d-4566-bf40-2732890754d6\") " Oct 30 21:57:12.794624 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:12.794542 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c04ccfd-188d-4566-bf40-2732890754d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c04ccfd-188d-4566-bf40-2732890754d6-workload-socket\") pod \"6c04ccfd-188d-4566-bf40-2732890754d6\" (UID: \"6c04ccfd-188d-4566-bf40-2732890754d6\") " Oct 30 21:57:12.794691 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:12.794604 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c04ccfd-188d-4566-bf40-2732890754d6-credential-socket" pod "6c04ccfd-188d-4566-bf40-2732890754d6" (UID: "6c04ccfd-188d-4566-bf40-2732890754d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:12.794691 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:12.794650 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c04ccfd-188d-4566-bf40-2732890754d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c04ccfd-188d-4566-bf40-2732890754d6-istio-podinfo\") pod \"6c04ccfd-188d-4566-bf40-2732890754d6\" (UID: \"6c04ccfd-188d-4566-bf40-2732890754d6\") " Oct 30 21:57:12.794691 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:12.794664 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c04ccfd-188d-4566-bf40-2732890754d6-workload-certs" pod "6c04ccfd-188d-4566-bf40-2732890754d6" (UID: "6c04ccfd-188d-4566-bf40-2732890754d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:12.794862 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:12.794697 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c04ccfd-188d-4566-bf40-2732890754d6-istio-data" pod "6c04ccfd-188d-4566-bf40-2732890754d6" (UID: "6c04ccfd-188d-4566-bf40-2732890754d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:12.794862 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:12.794855 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c04ccfd-188d-4566-bf40-2732890754d6-workload-socket" pod "6c04ccfd-188d-4566-bf40-2732890754d6" (UID: "6c04ccfd-188d-4566-bf40-2732890754d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:12.795352 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:12.795330 5748 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c04ccfd-188d-4566-bf40-2732890754d6-credential-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:12.795467 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:12.795356 5748 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c04ccfd-188d-4566-bf40-2732890754d6-workload-certs\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:12.795467 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:12.795370 5748 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c04ccfd-188d-4566-bf40-2732890754d6-istio-data\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:12.795467 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:12.795383 5748 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c04ccfd-188d-4566-bf40-2732890754d6-workload-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:12.795467 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:12.795398 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c04ccfd-188d-4566-bf40-2732890754d6-istiod-ca-cert" pod "6c04ccfd-188d-4566-bf40-2732890754d6" (UID: "6c04ccfd-188d-4566-bf40-2732890754d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:12.796582 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:12.796557 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c04ccfd-188d-4566-bf40-2732890754d6-istio-token" pod "6c04ccfd-188d-4566-bf40-2732890754d6" (UID: "6c04ccfd-188d-4566-bf40-2732890754d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:12.796691 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:12.796590 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c04ccfd-188d-4566-bf40-2732890754d6-kube-api-access-kzs59" pod "6c04ccfd-188d-4566-bf40-2732890754d6" (UID: "6c04ccfd-188d-4566-bf40-2732890754d6"). InnerVolumeSpecName "kube-api-access-kzs59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:12.796691 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:12.796664 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c04ccfd-188d-4566-bf40-2732890754d6-istio-envoy" pod "6c04ccfd-188d-4566-bf40-2732890754d6" (UID: "6c04ccfd-188d-4566-bf40-2732890754d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:12.796838 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:12.796763 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c04ccfd-188d-4566-bf40-2732890754d6-istio-podinfo" pod "6c04ccfd-188d-4566-bf40-2732890754d6" (UID: "6c04ccfd-188d-4566-bf40-2732890754d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:57:12.896110 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:12.896083 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5da50f53-661c-4e80-b55b-4ae2ab155ceb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5da50f53-661c-4e80-b55b-4ae2ab155ceb-maas-api-openshift-ca\") pod \"5da50f53-661c-4e80-b55b-4ae2ab155ceb\" (UID: \"5da50f53-661c-4e80-b55b-4ae2ab155ceb\") " Oct 30 21:57:12.896275 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:12.896171 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5da50f53-661c-4e80-b55b-4ae2ab155ceb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5da50f53-661c-4e80-b55b-4ae2ab155ceb-maas-api-xks-ca\") pod \"5da50f53-661c-4e80-b55b-4ae2ab155ceb\" (UID: \"5da50f53-661c-4e80-b55b-4ae2ab155ceb\") " Oct 30 21:57:12.896343 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:12.896307 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5da50f53-661c-4e80-b55b-4ae2ab155ceb-kube-api-access-wgt66\" (UniqueName: \"kubernetes.io/projected/5da50f53-661c-4e80-b55b-4ae2ab155ceb-kube-api-access-wgt66\") pod \"5da50f53-661c-4e80-b55b-4ae2ab155ceb\" (UID: \"5da50f53-661c-4e80-b55b-4ae2ab155ceb\") " Oct 30 21:57:12.896397 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:12.896380 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5da50f53-661c-4e80-b55b-4ae2ab155ceb-maas-api-openshift-ca" pod "5da50f53-661c-4e80-b55b-4ae2ab155ceb" (UID: "5da50f53-661c-4e80-b55b-4ae2ab155ceb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:12.896734 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:12.896715 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5da50f53-661c-4e80-b55b-4ae2ab155ceb-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:12.896814 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:12.896739 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzs59\" (UniqueName: \"kubernetes.io/projected/6c04ccfd-188d-4566-bf40-2732890754d6-kube-api-access-kzs59\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:12.896814 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:12.896754 5748 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c04ccfd-188d-4566-bf40-2732890754d6-istiod-ca-cert\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:12.896814 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:12.896767 5748 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c04ccfd-188d-4566-bf40-2732890754d6-istio-envoy\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:12.896814 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:12.896780 5748 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c04ccfd-188d-4566-bf40-2732890754d6-istio-podinfo\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:12.896814 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:12.896794 5748 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c04ccfd-188d-4566-bf40-2732890754d6-istio-token\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:12.898050 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:12.898023 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5da50f53-661c-4e80-b55b-4ae2ab155ceb-maas-api-xks-ca" pod "5da50f53-661c-4e80-b55b-4ae2ab155ceb" (UID: "5da50f53-661c-4e80-b55b-4ae2ab155ceb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:57:12.898150 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:12.898130 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5da50f53-661c-4e80-b55b-4ae2ab155ceb-kube-api-access-wgt66" pod "5da50f53-661c-4e80-b55b-4ae2ab155ceb" (UID: "5da50f53-661c-4e80-b55b-4ae2ab155ceb"). InnerVolumeSpecName "kube-api-access-wgt66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:12.997921 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:12.997823 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5da50f53-661c-4e80-b55b-4ae2ab155ceb-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:12.997921 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:12.997853 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgt66\" (UniqueName: \"kubernetes.io/projected/5da50f53-661c-4e80-b55b-4ae2ab155ceb-kube-api-access-wgt66\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:13.607131 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:13.607097 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e51c52-l8pv8" Oct 30 21:57:13.607131 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:13.607112 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e51c52-l8pv8" event={"ID":"5da50f53-661c-4e80-b55b-4ae2ab155ceb","Type":"ContainerDied","Data":"e176ff14190669edd60cee22b7296e50a57fd8db0de4960cc3439f71f3483675"} Oct 30 21:57:13.607621 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:13.607162 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e176ff14190669edd60cee22b7296e50a57fd8db0de4960cc3439f71f3483675" Oct 30 21:57:13.608600 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:13.608581 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bxj6r" Oct 30 21:57:13.608702 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:13.608617 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bxj6r" event={"ID":"6c04ccfd-188d-4566-bf40-2732890754d6","Type":"ContainerDied","Data":"93759c149a13e31810fd7bb0cdff193b67d007083530ffb4f427cb2c509863ec"} Oct 30 21:57:13.608702 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:13.608674 5748 scope.go:122] "RemoveContainer" containerID="4dde1979ea38af11d2d62f454c53385cee62f20bcad15fb3c9a84b69fd4727cf" Oct 30 21:57:13.777435 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:13.777359 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bxj6r"] Oct 30 21:57:13.827173 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:13.827140 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bxj6r"] Oct 30 21:57:14.468501 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:14.467744 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c04ccfd-188d-4566-bf40-2732890754d6" path="/var/lib/kubelet/pods/6c04ccfd-188d-4566-bf40-2732890754d6/volumes" Oct 30 21:57:24.077211 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:24.077175 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c92783-7b9gd"] Oct 30 21:57:24.077788 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:24.077773 5748 cpu_manager.go:414] "containerMap: removing container" podUID="5da50f53-661c-4e80-b55b-4ae2ab155ceb" containerName="revoke-keys" Oct 30 21:57:24.077837 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:24.077790 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5da50f53-661c-4e80-b55b-4ae2ab155ceb" containerName="revoke-keys" Oct 30 21:57:24.077837 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:24.077824 5748 cpu_manager.go:414] "containerMap: removing container" podUID="6c04ccfd-188d-4566-bf40-2732890754d6" containerName="istio-proxy" Oct 30 21:57:24.077837 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:24.077829 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c04ccfd-188d-4566-bf40-2732890754d6" containerName="istio-proxy" Oct 30 21:57:24.077965 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:24.077956 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="5da50f53-661c-4e80-b55b-4ae2ab155ceb" containerName="revoke-keys" Oct 30 21:57:24.078003 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:24.077969 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c04ccfd-188d-4566-bf40-2732890754d6" containerName="istio-proxy" Oct 30 21:57:24.082982 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:24.082965 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c92783-7b9gd" Oct 30 21:57:24.156918 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:24.149652 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c92783-7b9gd"] Oct 30 21:57:24.191408 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:24.191365 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 21:57:24.191839 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:24.191821 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:57:24.192118 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:24.192101 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:57:24.192401 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:24.192375 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:57:24.195425 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:24.195402 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15744eac-1e0a-49ba-892b-3425b609afc4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-c92783-7b9gd\" (UID: \"15744eac-1e0a-49ba-892b-3425b609afc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c92783-7b9gd" Oct 30 21:57:24.195661 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:24.195645 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z42s\" (UniqueName: \"kubernetes.io/projected/15744eac-1e0a-49ba-892b-3425b609afc4-kube-api-access-7z42s\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-c92783-7b9gd\" (UID: \"15744eac-1e0a-49ba-892b-3425b609afc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c92783-7b9gd" Oct 30 21:57:24.196037 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:24.196006 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15744eac-1e0a-49ba-892b-3425b609afc4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-c92783-7b9gd\" (UID: \"15744eac-1e0a-49ba-892b-3425b609afc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c92783-7b9gd" Oct 30 21:57:24.298623 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:24.298583 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z42s\" (UniqueName: \"kubernetes.io/projected/15744eac-1e0a-49ba-892b-3425b609afc4-kube-api-access-7z42s\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-c92783-7b9gd\" (UID: \"15744eac-1e0a-49ba-892b-3425b609afc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c92783-7b9gd" Oct 30 21:57:24.299042 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:24.299016 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15744eac-1e0a-49ba-892b-3425b609afc4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-c92783-7b9gd\" (UID: \"15744eac-1e0a-49ba-892b-3425b609afc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c92783-7b9gd" Oct 30 21:57:24.299300 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:24.299280 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15744eac-1e0a-49ba-892b-3425b609afc4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-c92783-7b9gd\" (UID: \"15744eac-1e0a-49ba-892b-3425b609afc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c92783-7b9gd" Oct 30 21:57:24.308920 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:24.300118 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15744eac-1e0a-49ba-892b-3425b609afc4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-c92783-7b9gd\" (UID: \"15744eac-1e0a-49ba-892b-3425b609afc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c92783-7b9gd" Oct 30 21:57:24.383796 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:24.383715 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15744eac-1e0a-49ba-892b-3425b609afc4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-c92783-7b9gd\" (UID: \"15744eac-1e0a-49ba-892b-3425b609afc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c92783-7b9gd" Oct 30 21:57:24.387765 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:24.387738 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z42s\" (UniqueName: \"kubernetes.io/projected/15744eac-1e0a-49ba-892b-3425b609afc4-kube-api-access-7z42s\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-c92783-7b9gd\" (UID: \"15744eac-1e0a-49ba-892b-3425b609afc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c92783-7b9gd" Oct 30 21:57:24.397730 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:24.397709 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c92783-7b9gd" Oct 30 21:57:25.129207 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:25.127298 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c92783-7b9gd"] Oct 30 21:57:25.741937 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:25.741879 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c92783-7b9gd" event={"ID":"15744eac-1e0a-49ba-892b-3425b609afc4","Type":"ContainerStarted","Data":"48e28b1566427a8be6d0d320a258b03d26d24fa3a229b5e3edc97ffb9f1a5173"} Oct 30 21:57:25.742111 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:25.741990 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c92783-7b9gd" event={"ID":"15744eac-1e0a-49ba-892b-3425b609afc4","Type":"ContainerStarted","Data":"8666dcea7efc488b2ba54c9dbad642eb88adebfa07a2a1a726c877c9ed8e5bf7"} Oct 30 21:57:25.862558 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:25.862448 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c92783-7b9gd" podStartSLOduration=1.8624340990000001 podStartE2EDuration="1.862434099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:25.861157157 +0000 UTC m=+2029.603678093" watchObservedRunningTime="2026-10-30 21:57:25.862434099 +0000 UTC m=+2029.604955033" Oct 30 21:57:26.750364 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:26.750324 5748 generic.go:356] "Generic (PLEG): container finished" podID="15744eac-1e0a-49ba-892b-3425b609afc4" containerID="48e28b1566427a8be6d0d320a258b03d26d24fa3a229b5e3edc97ffb9f1a5173" exitCode=0 Oct 30 21:57:26.750816 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:26.750382 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c92783-7b9gd" event={"ID":"15744eac-1e0a-49ba-892b-3425b609afc4","Type":"ContainerDied","Data":"48e28b1566427a8be6d0d320a258b03d26d24fa3a229b5e3edc97ffb9f1a5173"} Oct 30 21:57:27.912744 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:27.912708 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c92783-7b9gd" Oct 30 21:57:27.992749 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:27.992716 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15744eac-1e0a-49ba-892b-3425b609afc4-kube-api-access-7z42s\" (UniqueName: \"kubernetes.io/projected/15744eac-1e0a-49ba-892b-3425b609afc4-kube-api-access-7z42s\") pod \"15744eac-1e0a-49ba-892b-3425b609afc4\" (UID: \"15744eac-1e0a-49ba-892b-3425b609afc4\") " Oct 30 21:57:27.992938 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:27.992781 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15744eac-1e0a-49ba-892b-3425b609afc4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15744eac-1e0a-49ba-892b-3425b609afc4-maas-api-openshift-ca\") pod \"15744eac-1e0a-49ba-892b-3425b609afc4\" (UID: \"15744eac-1e0a-49ba-892b-3425b609afc4\") " Oct 30 21:57:27.993026 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:27.992977 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15744eac-1e0a-49ba-892b-3425b609afc4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15744eac-1e0a-49ba-892b-3425b609afc4-maas-api-xks-ca\") pod \"15744eac-1e0a-49ba-892b-3425b609afc4\" (UID: \"15744eac-1e0a-49ba-892b-3425b609afc4\") " Oct 30 21:57:27.993131 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:27.993108 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15744eac-1e0a-49ba-892b-3425b609afc4-maas-api-openshift-ca" pod "15744eac-1e0a-49ba-892b-3425b609afc4" (UID: "15744eac-1e0a-49ba-892b-3425b609afc4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:27.993429 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:27.993405 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15744eac-1e0a-49ba-892b-3425b609afc4-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:27.994942 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:27.994885 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15744eac-1e0a-49ba-892b-3425b609afc4-kube-api-access-7z42s" pod "15744eac-1e0a-49ba-892b-3425b609afc4" (UID: "15744eac-1e0a-49ba-892b-3425b609afc4"). InnerVolumeSpecName "kube-api-access-7z42s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:27.995078 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:27.995054 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15744eac-1e0a-49ba-892b-3425b609afc4-maas-api-xks-ca" pod "15744eac-1e0a-49ba-892b-3425b609afc4" (UID: "15744eac-1e0a-49ba-892b-3425b609afc4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:57:28.094980 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:28.094882 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7z42s\" (UniqueName: \"kubernetes.io/projected/15744eac-1e0a-49ba-892b-3425b609afc4-kube-api-access-7z42s\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:28.094980 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:28.094941 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15744eac-1e0a-49ba-892b-3425b609afc4-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:28.767415 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:28.767335 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c92783-7b9gd" Oct 30 21:57:28.767415 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:28.767347 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c92783-7b9gd" event={"ID":"15744eac-1e0a-49ba-892b-3425b609afc4","Type":"ContainerDied","Data":"8666dcea7efc488b2ba54c9dbad642eb88adebfa07a2a1a726c877c9ed8e5bf7"} Oct 30 21:57:28.767415 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:28.767393 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8666dcea7efc488b2ba54c9dbad642eb88adebfa07a2a1a726c877c9ed8e5bf7" Oct 30 21:57:30.145562 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:57:30.145521 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:57:31.792183 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:31.792149 5748 generic.go:356] "Generic (PLEG): container finished" podID="1faeab78-65f0-489f-9aeb-d651dd7cac1b" containerID="afa4d973d6d47fbdd9c9b149a638994e3606321fc430e0e577c5e9ab84c942c9" exitCode=0 Oct 30 21:57:31.792626 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:31.792223 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5" event={"ID":"1faeab78-65f0-489f-9aeb-d651dd7cac1b","Type":"ContainerDied","Data":"afa4d973d6d47fbdd9c9b149a638994e3606321fc430e0e577c5e9ab84c942c9"} Oct 30 21:57:31.792676 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:31.792662 5748 scope.go:122] "RemoveContainer" containerID="afa4d973d6d47fbdd9c9b149a638994e3606321fc430e0e577c5e9ab84c942c9" Oct 30 21:57:31.794683 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:31.794645 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:57:31.794790 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:31.794742 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:57:31.794830 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:31.794791 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:57:32.820920 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:32.800136 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5" event={"ID":"1faeab78-65f0-489f-9aeb-d651dd7cac1b","Type":"ContainerStarted","Data":"7161b894c7d64d0734bbfca6ed030cf9f8bcd31da3cbedddde8276f67088db2c"} Oct 30 21:57:33.239376 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:33.239346 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5" Oct 30 21:57:33.239595 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:33.239512 5748 patch_prober.go:29] interesting pod/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 21:57:33.239595 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:33.239555 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5" podUID="1faeab78-65f0-489f-9aeb-d651dd7cac1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 21:57:34.240504 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:34.240457 5748 patch_prober.go:29] interesting pod/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 21:57:34.241043 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:34.240531 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5" podUID="1faeab78-65f0-489f-9aeb-d651dd7cac1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 21:57:35.240921 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:35.239918 5748 patch_prober.go:29] interesting pod/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 21:57:35.240921 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:35.239995 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5" podUID="1faeab78-65f0-489f-9aeb-d651dd7cac1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 21:57:36.243455 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:36.243412 5748 patch_prober.go:29] interesting pod/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 21:57:36.244073 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:36.244035 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5" podUID="1faeab78-65f0-489f-9aeb-d651dd7cac1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 21:57:37.240278 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:37.240240 5748 patch_prober.go:29] interesting pod/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 21:57:37.240487 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:37.240309 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5" podUID="1faeab78-65f0-489f-9aeb-d651dd7cac1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 21:57:37.532592 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:37.532499 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-948nx"] Oct 30 21:57:37.534429 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:37.534399 5748 cpu_manager.go:414] "containerMap: removing container" podUID="15744eac-1e0a-49ba-892b-3425b609afc4" containerName="revoke-keys" Oct 30 21:57:37.534600 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:37.534585 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15744eac-1e0a-49ba-892b-3425b609afc4" containerName="revoke-keys" Oct 30 21:57:37.535094 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:37.535073 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="15744eac-1e0a-49ba-892b-3425b609afc4" containerName="revoke-keys" Oct 30 21:57:37.539188 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:37.539168 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-948nx" Oct 30 21:57:37.583872 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:37.579881 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-79e10a-rqj7g"] Oct 30 21:57:37.583872 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:37.582573 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-79e10a-rqj7g"] Oct 30 21:57:37.640974 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:37.640943 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-948nx"] Oct 30 21:57:37.660573 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:37.660540 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f411f535-b1bc-408b-afb9-3299fc9d5bd6-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-948nx\" (UID: \"f411f535-b1bc-408b-afb9-3299fc9d5bd6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-948nx" Oct 30 21:57:37.660748 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:37.660610 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f411f535-b1bc-408b-afb9-3299fc9d5bd6-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-948nx\" (UID: \"f411f535-b1bc-408b-afb9-3299fc9d5bd6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-948nx" Oct 30 21:57:37.660748 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:37.660719 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f411f535-b1bc-408b-afb9-3299fc9d5bd6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-948nx\" (UID: \"f411f535-b1bc-408b-afb9-3299fc9d5bd6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-948nx" Oct 30 21:57:37.660869 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:37.660750 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f411f535-b1bc-408b-afb9-3299fc9d5bd6-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-948nx\" (UID: \"f411f535-b1bc-408b-afb9-3299fc9d5bd6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-948nx" Oct 30 21:57:37.660943 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:37.660872 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f411f535-b1bc-408b-afb9-3299fc9d5bd6-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-948nx\" (UID: \"f411f535-b1bc-408b-afb9-3299fc9d5bd6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-948nx" Oct 30 21:57:37.661008 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:37.660969 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f411f535-b1bc-408b-afb9-3299fc9d5bd6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-948nx\" (UID: \"f411f535-b1bc-408b-afb9-3299fc9d5bd6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-948nx" Oct 30 21:57:37.661008 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:37.660998 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f411f535-b1bc-408b-afb9-3299fc9d5bd6-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-948nx\" (UID: \"f411f535-b1bc-408b-afb9-3299fc9d5bd6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-948nx" Oct 30 21:57:37.661113 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:37.661035 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f411f535-b1bc-408b-afb9-3299fc9d5bd6-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-948nx\" (UID: \"f411f535-b1bc-408b-afb9-3299fc9d5bd6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-948nx" Oct 30 21:57:37.661167 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:37.661146 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzxtl\" (UniqueName: \"kubernetes.io/projected/f411f535-b1bc-408b-afb9-3299fc9d5bd6-kube-api-access-qzxtl\") pod \"maas-default-gateway-openshift-default-75d7c785f-948nx\" (UID: \"f411f535-b1bc-408b-afb9-3299fc9d5bd6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-948nx" Oct 30 21:57:37.688830 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:37.688801 5748 scope.go:122] "RemoveContainer" containerID="9fae17f77b9fdead800555a8766dbd87f37d02a5af6d679fc193ad021a68a5a3" Oct 30 21:57:37.698733 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:37.698714 5748 scope.go:122] "RemoveContainer" containerID="e43a96f6d8f02150bf2f68c1a92047f197e8aadc1eb516859d1a2f446e005f6d" Oct 30 21:57:37.763653 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:37.763629 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f411f535-b1bc-408b-afb9-3299fc9d5bd6-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-948nx\" (UID: \"f411f535-b1bc-408b-afb9-3299fc9d5bd6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-948nx" Oct 30 21:57:37.763740 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:37.763698 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f411f535-b1bc-408b-afb9-3299fc9d5bd6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-948nx\" (UID: \"f411f535-b1bc-408b-afb9-3299fc9d5bd6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-948nx" Oct 30 21:57:37.763740 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:37.763728 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f411f535-b1bc-408b-afb9-3299fc9d5bd6-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-948nx\" (UID: \"f411f535-b1bc-408b-afb9-3299fc9d5bd6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-948nx" Oct 30 21:57:37.763864 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:37.763785 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f411f535-b1bc-408b-afb9-3299fc9d5bd6-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-948nx\" (UID: \"f411f535-b1bc-408b-afb9-3299fc9d5bd6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-948nx" Oct 30 21:57:37.763864 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:37.763851 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzxtl\" (UniqueName: \"kubernetes.io/projected/f411f535-b1bc-408b-afb9-3299fc9d5bd6-kube-api-access-qzxtl\") pod \"maas-default-gateway-openshift-default-75d7c785f-948nx\" (UID: \"f411f535-b1bc-408b-afb9-3299fc9d5bd6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-948nx" Oct 30 21:57:37.764068 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:37.764046 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f411f535-b1bc-408b-afb9-3299fc9d5bd6-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-948nx\" (UID: \"f411f535-b1bc-408b-afb9-3299fc9d5bd6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-948nx" Oct 30 21:57:37.764197 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:37.764180 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f411f535-b1bc-408b-afb9-3299fc9d5bd6-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-948nx\" (UID: \"f411f535-b1bc-408b-afb9-3299fc9d5bd6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-948nx" Oct 30 21:57:37.764261 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:37.764202 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f411f535-b1bc-408b-afb9-3299fc9d5bd6-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-948nx\" (UID: \"f411f535-b1bc-408b-afb9-3299fc9d5bd6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-948nx" Oct 30 21:57:37.764322 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:37.764273 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f411f535-b1bc-408b-afb9-3299fc9d5bd6-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-948nx\" (UID: \"f411f535-b1bc-408b-afb9-3299fc9d5bd6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-948nx" Oct 30 21:57:37.764397 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:37.764382 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f411f535-b1bc-408b-afb9-3299fc9d5bd6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-948nx\" (UID: \"f411f535-b1bc-408b-afb9-3299fc9d5bd6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-948nx" Oct 30 21:57:37.764454 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:37.764430 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f411f535-b1bc-408b-afb9-3299fc9d5bd6-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-948nx\" (UID: \"f411f535-b1bc-408b-afb9-3299fc9d5bd6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-948nx" Oct 30 21:57:37.764513 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:37.764452 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f411f535-b1bc-408b-afb9-3299fc9d5bd6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-948nx\" (UID: \"f411f535-b1bc-408b-afb9-3299fc9d5bd6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-948nx" Oct 30 21:57:37.764639 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:37.764611 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f411f535-b1bc-408b-afb9-3299fc9d5bd6-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-948nx\" (UID: \"f411f535-b1bc-408b-afb9-3299fc9d5bd6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-948nx" Oct 30 21:57:37.764802 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:37.764772 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f411f535-b1bc-408b-afb9-3299fc9d5bd6-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-948nx\" (UID: \"f411f535-b1bc-408b-afb9-3299fc9d5bd6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-948nx" Oct 30 21:57:37.766109 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:37.766086 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f411f535-b1bc-408b-afb9-3299fc9d5bd6-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-948nx\" (UID: \"f411f535-b1bc-408b-afb9-3299fc9d5bd6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-948nx" Oct 30 21:57:37.766630 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:37.766612 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f411f535-b1bc-408b-afb9-3299fc9d5bd6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-948nx\" (UID: \"f411f535-b1bc-408b-afb9-3299fc9d5bd6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-948nx" Oct 30 21:57:37.827067 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:37.827007 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzxtl\" (UniqueName: \"kubernetes.io/projected/f411f535-b1bc-408b-afb9-3299fc9d5bd6-kube-api-access-qzxtl\") pod \"maas-default-gateway-openshift-default-75d7c785f-948nx\" (UID: \"f411f535-b1bc-408b-afb9-3299fc9d5bd6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-948nx" Oct 30 21:57:37.827240 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:37.827123 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f411f535-b1bc-408b-afb9-3299fc9d5bd6-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-948nx\" (UID: \"f411f535-b1bc-408b-afb9-3299fc9d5bd6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-948nx" Oct 30 21:57:37.855498 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:37.855476 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-948nx" Oct 30 21:57:38.284784 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:38.284751 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5" Oct 30 21:57:38.285315 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:38.285289 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5" Oct 30 21:57:38.286573 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:38.286551 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5" Oct 30 21:57:38.313976 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:38.313948 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-948nx"] Oct 30 21:57:38.314768 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:57:38.314743 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf411f535_b1bc_408b_afb9_3299fc9d5bd6.slice/crio-5256b328b603e266b37b9fd59e0e16d226bb07fcaa2271dc48906ceb227a85ce WatchSource:0}: Error finding container 5256b328b603e266b37b9fd59e0e16d226bb07fcaa2271dc48906ceb227a85ce: Status 404 returned error can't find the container with id 5256b328b603e266b37b9fd59e0e16d226bb07fcaa2271dc48906ceb227a85ce Oct 30 21:57:38.317380 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:38.317346 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:57:38.317485 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:38.317423 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:57:38.317485 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:38.317469 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:57:38.418668 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:38.418628 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85e4ecda-eb68-4c06-abb0-b6b8ea5db24c" path="/var/lib/kubelet/pods/85e4ecda-eb68-4c06-abb0-b6b8ea5db24c/volumes" Oct 30 21:57:38.936703 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:38.936666 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-948nx" event={"ID":"f411f535-b1bc-408b-afb9-3299fc9d5bd6","Type":"ContainerStarted","Data":"86ea75b538cb5dfa3cd2d1c6a3d88361eb02d692fce95392106c5475ea6605f2"} Oct 30 21:57:38.937150 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:38.936731 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-948nx" event={"ID":"f411f535-b1bc-408b-afb9-3299fc9d5bd6","Type":"ContainerStarted","Data":"5256b328b603e266b37b9fd59e0e16d226bb07fcaa2271dc48906ceb227a85ce"} Oct 30 21:57:39.059962 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:39.059891 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-948nx" podStartSLOduration=2.059872137 podStartE2EDuration="2.059872137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:39.058487448 +0000 UTC m=+2042.801008385" watchObservedRunningTime="2026-10-30 21:57:39.059872137 +0000 UTC m=+2042.802393071" Oct 30 21:57:39.856190 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:39.856158 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-948nx" Oct 30 21:57:39.857448 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:39.857420 5748 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-948nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 30 21:57:39.857588 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:39.857463 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-948nx" podUID="f411f535-b1bc-408b-afb9-3299fc9d5bd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 30 21:57:40.856050 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:40.856014 5748 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-948nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 30 21:57:40.856468 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:40.856073 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-948nx" podUID="f411f535-b1bc-408b-afb9-3299fc9d5bd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 30 21:57:41.856167 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:41.856134 5748 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-948nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 30 21:57:41.856659 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:41.856184 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-948nx" podUID="f411f535-b1bc-408b-afb9-3299fc9d5bd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 30 21:57:42.147424 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:42.147389 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d6e0be-pmzv7"] Oct 30 21:57:42.150447 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:42.150428 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d6e0be-pmzv7" Oct 30 21:57:42.222919 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:42.222085 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 21:57:42.222919 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:42.222777 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:57:42.230919 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:42.230762 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d6e0be-pmzv7"] Oct 30 21:57:42.265534 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:42.265490 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e3b179f-4f16-45e8-8f6d-8be062edf9f3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-d6e0be-pmzv7\" (UID: \"2e3b179f-4f16-45e8-8f6d-8be062edf9f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d6e0be-pmzv7" Oct 30 21:57:42.265695 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:42.265622 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gstkj\" (UniqueName: \"kubernetes.io/projected/2e3b179f-4f16-45e8-8f6d-8be062edf9f3-kube-api-access-gstkj\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-d6e0be-pmzv7\" (UID: \"2e3b179f-4f16-45e8-8f6d-8be062edf9f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d6e0be-pmzv7" Oct 30 21:57:42.265764 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:42.265728 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:57:42.265764 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:42.265742 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:57:42.265875 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:42.265780 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e3b179f-4f16-45e8-8f6d-8be062edf9f3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-d6e0be-pmzv7\" (UID: \"2e3b179f-4f16-45e8-8f6d-8be062edf9f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d6e0be-pmzv7" Oct 30 21:57:42.367230 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:42.367199 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gstkj\" (UniqueName: \"kubernetes.io/projected/2e3b179f-4f16-45e8-8f6d-8be062edf9f3-kube-api-access-gstkj\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-d6e0be-pmzv7\" (UID: \"2e3b179f-4f16-45e8-8f6d-8be062edf9f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d6e0be-pmzv7" Oct 30 21:57:42.367384 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:42.367323 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e3b179f-4f16-45e8-8f6d-8be062edf9f3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-d6e0be-pmzv7\" (UID: \"2e3b179f-4f16-45e8-8f6d-8be062edf9f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d6e0be-pmzv7" Oct 30 21:57:42.367384 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:42.367370 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e3b179f-4f16-45e8-8f6d-8be062edf9f3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-d6e0be-pmzv7\" (UID: \"2e3b179f-4f16-45e8-8f6d-8be062edf9f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d6e0be-pmzv7" Oct 30 21:57:42.368098 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:42.368075 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e3b179f-4f16-45e8-8f6d-8be062edf9f3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-d6e0be-pmzv7\" (UID: \"2e3b179f-4f16-45e8-8f6d-8be062edf9f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d6e0be-pmzv7" Oct 30 21:57:42.369669 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:42.369645 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e3b179f-4f16-45e8-8f6d-8be062edf9f3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-d6e0be-pmzv7\" (UID: \"2e3b179f-4f16-45e8-8f6d-8be062edf9f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d6e0be-pmzv7" Oct 30 21:57:42.429475 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:42.429399 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gstkj\" (UniqueName: \"kubernetes.io/projected/2e3b179f-4f16-45e8-8f6d-8be062edf9f3-kube-api-access-gstkj\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-d6e0be-pmzv7\" (UID: \"2e3b179f-4f16-45e8-8f6d-8be062edf9f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d6e0be-pmzv7" Oct 30 21:57:42.464563 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:42.464531 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d6e0be-pmzv7" Oct 30 21:57:42.856283 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:42.856199 5748 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-948nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 30 21:57:42.856283 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:42.856253 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-948nx" podUID="f411f535-b1bc-408b-afb9-3299fc9d5bd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 30 21:57:42.876208 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:42.876178 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d6e0be-pmzv7"] Oct 30 21:57:42.877649 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:57:42.877619 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e3b179f_4f16_45e8_8f6d_8be062edf9f3.slice/crio-4e78045495d63af0fb647b00383470eb30fc7b96a888e68c944a9375a83a7372 WatchSource:0}: Error finding container 4e78045495d63af0fb647b00383470eb30fc7b96a888e68c944a9375a83a7372: Status 404 returned error can't find the container with id 4e78045495d63af0fb647b00383470eb30fc7b96a888e68c944a9375a83a7372 Oct 30 21:57:42.963031 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:42.962996 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d6e0be-pmzv7" event={"ID":"2e3b179f-4f16-45e8-8f6d-8be062edf9f3","Type":"ContainerStarted","Data":"4e78045495d63af0fb647b00383470eb30fc7b96a888e68c944a9375a83a7372"} Oct 30 21:57:43.856756 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:43.856724 5748 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-948nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 30 21:57:43.857189 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:43.856777 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-948nx" podUID="f411f535-b1bc-408b-afb9-3299fc9d5bd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 30 21:57:43.970547 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:43.970507 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d6e0be-pmzv7" event={"ID":"2e3b179f-4f16-45e8-8f6d-8be062edf9f3","Type":"ContainerStarted","Data":"d81d9e5274309ca1cd2101a5f2c5a668aa9c87ca31ffdb433438ddd7ed40da75"} Oct 30 21:57:44.089040 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:44.088992 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d6e0be-pmzv7" podStartSLOduration=2.08897652 podStartE2EDuration="2.08897652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:44.088220746 +0000 UTC m=+2047.830741681" watchObservedRunningTime="2026-10-30 21:57:44.08897652 +0000 UTC m=+2047.831497455" Oct 30 21:57:44.856641 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:44.856590 5748 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-948nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 30 21:57:44.856820 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:44.856660 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-948nx" podUID="f411f535-b1bc-408b-afb9-3299fc9d5bd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 30 21:57:44.936430 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:44.936395 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226"] Oct 30 21:57:44.936610 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:44.936587 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226" podUID="e9aab906-835e-4acf-b20f-ef27d8bd8568" containerName="istio-proxy" containerID="cri-o://9f4a7086c80be63b67796c15a89a6eb8fd2dc9d59045928508cbf8418b4ae59b" gracePeriod=30 Oct 30 21:57:44.978713 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:44.978674 5748 generic.go:356] "Generic (PLEG): container finished" podID="2e3b179f-4f16-45e8-8f6d-8be062edf9f3" containerID="d81d9e5274309ca1cd2101a5f2c5a668aa9c87ca31ffdb433438ddd7ed40da75" exitCode=0 Oct 30 21:57:44.978866 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:44.978738 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d6e0be-pmzv7" event={"ID":"2e3b179f-4f16-45e8-8f6d-8be062edf9f3","Type":"ContainerDied","Data":"d81d9e5274309ca1cd2101a5f2c5a668aa9c87ca31ffdb433438ddd7ed40da75"} Oct 30 21:57:45.856469 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:45.856428 5748 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-948nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 30 21:57:45.856654 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:45.856491 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-948nx" podUID="f411f535-b1bc-408b-afb9-3299fc9d5bd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 30 21:57:46.158622 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:46.158595 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d6e0be-pmzv7" Oct 30 21:57:46.224630 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:46.224602 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e3b179f-4f16-45e8-8f6d-8be062edf9f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e3b179f-4f16-45e8-8f6d-8be062edf9f3-maas-api-openshift-ca\") pod \"2e3b179f-4f16-45e8-8f6d-8be062edf9f3\" (UID: \"2e3b179f-4f16-45e8-8f6d-8be062edf9f3\") " Oct 30 21:57:46.224823 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:46.224760 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e3b179f-4f16-45e8-8f6d-8be062edf9f3-kube-api-access-gstkj\" (UniqueName: \"kubernetes.io/projected/2e3b179f-4f16-45e8-8f6d-8be062edf9f3-kube-api-access-gstkj\") pod \"2e3b179f-4f16-45e8-8f6d-8be062edf9f3\" (UID: \"2e3b179f-4f16-45e8-8f6d-8be062edf9f3\") " Oct 30 21:57:46.224823 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:46.224813 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e3b179f-4f16-45e8-8f6d-8be062edf9f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e3b179f-4f16-45e8-8f6d-8be062edf9f3-maas-api-xks-ca\") pod \"2e3b179f-4f16-45e8-8f6d-8be062edf9f3\" (UID: \"2e3b179f-4f16-45e8-8f6d-8be062edf9f3\") " Oct 30 21:57:46.225065 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:46.225041 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e3b179f-4f16-45e8-8f6d-8be062edf9f3-maas-api-openshift-ca" pod "2e3b179f-4f16-45e8-8f6d-8be062edf9f3" (UID: "2e3b179f-4f16-45e8-8f6d-8be062edf9f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:46.225341 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:46.225322 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e3b179f-4f16-45e8-8f6d-8be062edf9f3-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:46.227469 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:46.227440 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e3b179f-4f16-45e8-8f6d-8be062edf9f3-maas-api-xks-ca" pod "2e3b179f-4f16-45e8-8f6d-8be062edf9f3" (UID: "2e3b179f-4f16-45e8-8f6d-8be062edf9f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:57:46.227469 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:46.227465 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e3b179f-4f16-45e8-8f6d-8be062edf9f3-kube-api-access-gstkj" pod "2e3b179f-4f16-45e8-8f6d-8be062edf9f3" (UID: "2e3b179f-4f16-45e8-8f6d-8be062edf9f3"). InnerVolumeSpecName "kube-api-access-gstkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:46.327315 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:46.327275 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gstkj\" (UniqueName: \"kubernetes.io/projected/2e3b179f-4f16-45e8-8f6d-8be062edf9f3-kube-api-access-gstkj\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:46.327315 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:46.327317 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e3b179f-4f16-45e8-8f6d-8be062edf9f3-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:46.998735 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:46.998703 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d6e0be-pmzv7" Oct 30 21:57:46.998735 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:46.998714 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d6e0be-pmzv7" event={"ID":"2e3b179f-4f16-45e8-8f6d-8be062edf9f3","Type":"ContainerDied","Data":"4e78045495d63af0fb647b00383470eb30fc7b96a888e68c944a9375a83a7372"} Oct 30 21:57:46.998996 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:46.998758 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e78045495d63af0fb647b00383470eb30fc7b96a888e68c944a9375a83a7372" Oct 30 21:57:47.003275 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:47.003241 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-948nx" Oct 30 21:57:47.003684 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:47.003659 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-948nx" Oct 30 21:57:47.004476 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:47.004459 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-948nx" Oct 30 21:57:50.224485 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:50.224462 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226" Oct 30 21:57:50.281194 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:50.281156 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9aab906-835e-4acf-b20f-ef27d8bd8568-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9aab906-835e-4acf-b20f-ef27d8bd8568-credential-socket\") pod \"e9aab906-835e-4acf-b20f-ef27d8bd8568\" (UID: \"e9aab906-835e-4acf-b20f-ef27d8bd8568\") " Oct 30 21:57:50.281384 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:50.281279 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e9aab906-835e-4acf-b20f-ef27d8bd8568-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9aab906-835e-4acf-b20f-ef27d8bd8568-istio-podinfo\") pod \"e9aab906-835e-4acf-b20f-ef27d8bd8568\" (UID: \"e9aab906-835e-4acf-b20f-ef27d8bd8568\") " Oct 30 21:57:50.281384 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:50.281333 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9aab906-835e-4acf-b20f-ef27d8bd8568-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9aab906-835e-4acf-b20f-ef27d8bd8568-istio-envoy\") pod \"e9aab906-835e-4acf-b20f-ef27d8bd8568\" (UID: \"e9aab906-835e-4acf-b20f-ef27d8bd8568\") " Oct 30 21:57:50.281384 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:50.281368 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9aab906-835e-4acf-b20f-ef27d8bd8568-kube-api-access-nkplt\" (UniqueName: \"kubernetes.io/projected/e9aab906-835e-4acf-b20f-ef27d8bd8568-kube-api-access-nkplt\") pod \"e9aab906-835e-4acf-b20f-ef27d8bd8568\" (UID: \"e9aab906-835e-4acf-b20f-ef27d8bd8568\") " Oct 30 21:57:50.281569 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:50.281393 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9aab906-835e-4acf-b20f-ef27d8bd8568-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9aab906-835e-4acf-b20f-ef27d8bd8568-workload-socket\") pod \"e9aab906-835e-4acf-b20f-ef27d8bd8568\" (UID: \"e9aab906-835e-4acf-b20f-ef27d8bd8568\") " Oct 30 21:57:50.281569 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:50.281474 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9aab906-835e-4acf-b20f-ef27d8bd8568-istio-token\" (UniqueName: \"kubernetes.io/projected/e9aab906-835e-4acf-b20f-ef27d8bd8568-istio-token\") pod \"e9aab906-835e-4acf-b20f-ef27d8bd8568\" (UID: \"e9aab906-835e-4acf-b20f-ef27d8bd8568\") " Oct 30 21:57:50.281569 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:50.281485 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9aab906-835e-4acf-b20f-ef27d8bd8568-credential-socket" pod "e9aab906-835e-4acf-b20f-ef27d8bd8568" (UID: "e9aab906-835e-4acf-b20f-ef27d8bd8568"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:50.281569 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:50.281528 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9aab906-835e-4acf-b20f-ef27d8bd8568-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9aab906-835e-4acf-b20f-ef27d8bd8568-istio-data\") pod \"e9aab906-835e-4acf-b20f-ef27d8bd8568\" (UID: \"e9aab906-835e-4acf-b20f-ef27d8bd8568\") " Oct 30 21:57:50.281569 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:50.281564 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9aab906-835e-4acf-b20f-ef27d8bd8568-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9aab906-835e-4acf-b20f-ef27d8bd8568-workload-certs\") pod \"e9aab906-835e-4acf-b20f-ef27d8bd8568\" (UID: \"e9aab906-835e-4acf-b20f-ef27d8bd8568\") " Oct 30 21:57:50.281825 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:50.281597 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9aab906-835e-4acf-b20f-ef27d8bd8568-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9aab906-835e-4acf-b20f-ef27d8bd8568-istiod-ca-cert\") pod \"e9aab906-835e-4acf-b20f-ef27d8bd8568\" (UID: \"e9aab906-835e-4acf-b20f-ef27d8bd8568\") " Oct 30 21:57:50.281825 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:50.281746 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9aab906-835e-4acf-b20f-ef27d8bd8568-workload-socket" pod "e9aab906-835e-4acf-b20f-ef27d8bd8568" (UID: "e9aab906-835e-4acf-b20f-ef27d8bd8568"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:50.281970 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:50.281926 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9aab906-835e-4acf-b20f-ef27d8bd8568-istio-data" pod "e9aab906-835e-4acf-b20f-ef27d8bd8568" (UID: "e9aab906-835e-4acf-b20f-ef27d8bd8568"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:50.281970 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:50.281939 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9aab906-835e-4acf-b20f-ef27d8bd8568-workload-certs" pod "e9aab906-835e-4acf-b20f-ef27d8bd8568" (UID: "e9aab906-835e-4acf-b20f-ef27d8bd8568"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:50.282214 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:50.282185 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9aab906-835e-4acf-b20f-ef27d8bd8568-istiod-ca-cert" pod "e9aab906-835e-4acf-b20f-ef27d8bd8568" (UID: "e9aab906-835e-4acf-b20f-ef27d8bd8568"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:50.282427 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:50.282405 5748 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9aab906-835e-4acf-b20f-ef27d8bd8568-workload-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:50.282520 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:50.282436 5748 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9aab906-835e-4acf-b20f-ef27d8bd8568-istio-data\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:50.282520 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:50.282458 5748 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9aab906-835e-4acf-b20f-ef27d8bd8568-workload-certs\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:50.282520 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:50.282472 5748 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9aab906-835e-4acf-b20f-ef27d8bd8568-istiod-ca-cert\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:50.282520 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:50.282487 5748 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9aab906-835e-4acf-b20f-ef27d8bd8568-credential-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:50.283688 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:50.283661 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9aab906-835e-4acf-b20f-ef27d8bd8568-istio-podinfo" pod "e9aab906-835e-4acf-b20f-ef27d8bd8568" (UID: "e9aab906-835e-4acf-b20f-ef27d8bd8568"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:57:50.283931 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:50.283891 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9aab906-835e-4acf-b20f-ef27d8bd8568-istio-token" pod "e9aab906-835e-4acf-b20f-ef27d8bd8568" (UID: "e9aab906-835e-4acf-b20f-ef27d8bd8568"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:50.284047 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:50.284026 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9aab906-835e-4acf-b20f-ef27d8bd8568-kube-api-access-nkplt" pod "e9aab906-835e-4acf-b20f-ef27d8bd8568" (UID: "e9aab906-835e-4acf-b20f-ef27d8bd8568"). InnerVolumeSpecName "kube-api-access-nkplt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:50.284106 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:50.284057 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9aab906-835e-4acf-b20f-ef27d8bd8568-istio-envoy" pod "e9aab906-835e-4acf-b20f-ef27d8bd8568" (UID: "e9aab906-835e-4acf-b20f-ef27d8bd8568"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:50.384015 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:50.383985 5748 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9aab906-835e-4acf-b20f-ef27d8bd8568-istio-podinfo\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:50.384015 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:50.384012 5748 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9aab906-835e-4acf-b20f-ef27d8bd8568-istio-envoy\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:50.384189 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:50.384027 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkplt\" (UniqueName: \"kubernetes.io/projected/e9aab906-835e-4acf-b20f-ef27d8bd8568-kube-api-access-nkplt\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:50.384189 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:50.384037 5748 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9aab906-835e-4acf-b20f-ef27d8bd8568-istio-token\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:50.666765 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:50.666733 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-259e81-wbqvt"] Oct 30 21:57:50.669385 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:50.669361 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-259e81-wbqvt"] Oct 30 21:57:51.027791 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:51.027702 5748 generic.go:356] "Generic (PLEG): container finished" podID="e9aab906-835e-4acf-b20f-ef27d8bd8568" containerID="9f4a7086c80be63b67796c15a89a6eb8fd2dc9d59045928508cbf8418b4ae59b" exitCode=0 Oct 30 21:57:51.027791 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:51.027767 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226" Oct 30 21:57:51.028041 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:51.027787 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226" event={"ID":"e9aab906-835e-4acf-b20f-ef27d8bd8568","Type":"ContainerDied","Data":"9f4a7086c80be63b67796c15a89a6eb8fd2dc9d59045928508cbf8418b4ae59b"} Oct 30 21:57:51.028041 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:51.027844 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226" event={"ID":"e9aab906-835e-4acf-b20f-ef27d8bd8568","Type":"ContainerDied","Data":"90760c108a0714d9fee2863f1d69ca1edae0da79872cd730c8a3b5b712e3a739"} Oct 30 21:57:51.028041 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:51.027869 5748 scope.go:122] "RemoveContainer" containerID="9f4a7086c80be63b67796c15a89a6eb8fd2dc9d59045928508cbf8418b4ae59b" Oct 30 21:57:51.037829 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:51.037813 5748 scope.go:122] "RemoveContainer" containerID="36b317e896eca6413b3fcf645a68946f9e9ae4a601b45ca6f7be2a2c3504adfe" Oct 30 21:57:51.048011 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:51.047991 5748 scope.go:122] "RemoveContainer" containerID="9f4a7086c80be63b67796c15a89a6eb8fd2dc9d59045928508cbf8418b4ae59b" Oct 30 21:57:51.048321 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:57:51.048294 5748 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f4a7086c80be63b67796c15a89a6eb8fd2dc9d59045928508cbf8418b4ae59b\": container with ID starting with 9f4a7086c80be63b67796c15a89a6eb8fd2dc9d59045928508cbf8418b4ae59b not found: ID does not exist" containerID="9f4a7086c80be63b67796c15a89a6eb8fd2dc9d59045928508cbf8418b4ae59b" Oct 30 21:57:51.048437 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:51.048342 5748 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f4a7086c80be63b67796c15a89a6eb8fd2dc9d59045928508cbf8418b4ae59b"} err="failed to get container status \"9f4a7086c80be63b67796c15a89a6eb8fd2dc9d59045928508cbf8418b4ae59b\": rpc error: code = NotFound desc = could not find container \"9f4a7086c80be63b67796c15a89a6eb8fd2dc9d59045928508cbf8418b4ae59b\": container with ID starting with 9f4a7086c80be63b67796c15a89a6eb8fd2dc9d59045928508cbf8418b4ae59b not found: ID does not exist" Oct 30 21:57:51.048437 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:51.048361 5748 scope.go:122] "RemoveContainer" containerID="36b317e896eca6413b3fcf645a68946f9e9ae4a601b45ca6f7be2a2c3504adfe" Oct 30 21:57:51.048817 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:57:51.048786 5748 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36b317e896eca6413b3fcf645a68946f9e9ae4a601b45ca6f7be2a2c3504adfe\": container with ID starting with 36b317e896eca6413b3fcf645a68946f9e9ae4a601b45ca6f7be2a2c3504adfe not found: ID does not exist" containerID="36b317e896eca6413b3fcf645a68946f9e9ae4a601b45ca6f7be2a2c3504adfe" Oct 30 21:57:51.048951 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:51.048819 5748 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36b317e896eca6413b3fcf645a68946f9e9ae4a601b45ca6f7be2a2c3504adfe"} err="failed to get container status \"36b317e896eca6413b3fcf645a68946f9e9ae4a601b45ca6f7be2a2c3504adfe\": rpc error: code = NotFound desc = could not find container \"36b317e896eca6413b3fcf645a68946f9e9ae4a601b45ca6f7be2a2c3504adfe\": container with ID starting with 36b317e896eca6413b3fcf645a68946f9e9ae4a601b45ca6f7be2a2c3504adfe not found: ID does not exist" Oct 30 21:57:51.198648 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:51.198613 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226"] Oct 30 21:57:51.202759 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:51.202732 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-79e10a-openshift-default-7d55bb5987-5z226"] Oct 30 21:57:52.417190 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:52.417158 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b5371f1-01b6-4b96-ace9-52a566fd635c" path="/var/lib/kubelet/pods/6b5371f1-01b6-4b96-ace9-52a566fd635c/volumes" Oct 30 21:57:52.417660 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:57:52.417633 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9aab906-835e-4acf-b20f-ef27d8bd8568" path="/var/lib/kubelet/pods/e9aab906-835e-4acf-b20f-ef27d8bd8568/volumes" Oct 30 21:58:00.143826 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:58:00.143786 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:58:17.776234 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:17.776197 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71a3d788-6rwdh"] Oct 30 21:58:17.826506 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:17.826473 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71a3d788-6rwdh"] Oct 30 21:58:18.491444 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:18.491373 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e552129-6a1f-41a3-a8c0-3dd7512447a5" path="/var/lib/kubelet/pods/2e552129-6a1f-41a3-a8c0-3dd7512447a5/volumes" Oct 30 21:58:19.212878 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:19.212845 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-bd3c59-dmzp9"] Oct 30 21:58:19.213479 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:19.213464 5748 cpu_manager.go:414] "containerMap: removing container" podUID="e9aab906-835e-4acf-b20f-ef27d8bd8568" containerName="istio-proxy" Oct 30 21:58:19.213523 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:19.213480 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9aab906-835e-4acf-b20f-ef27d8bd8568" containerName="istio-proxy" Oct 30 21:58:19.213523 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:19.213493 5748 cpu_manager.go:414] "containerMap: removing container" podUID="e9aab906-835e-4acf-b20f-ef27d8bd8568" containerName="istio-proxy" Oct 30 21:58:19.213523 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:19.213498 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9aab906-835e-4acf-b20f-ef27d8bd8568" containerName="istio-proxy" Oct 30 21:58:19.213523 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:19.213514 5748 cpu_manager.go:414] "containerMap: removing container" podUID="2e3b179f-4f16-45e8-8f6d-8be062edf9f3" containerName="revoke-keys" Oct 30 21:58:19.213523 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:19.213519 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e3b179f-4f16-45e8-8f6d-8be062edf9f3" containerName="revoke-keys" Oct 30 21:58:19.213680 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:19.213606 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9aab906-835e-4acf-b20f-ef27d8bd8568" containerName="istio-proxy" Oct 30 21:58:19.213680 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:19.213614 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9aab906-835e-4acf-b20f-ef27d8bd8568" containerName="istio-proxy" Oct 30 21:58:19.213680 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:19.213624 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e3b179f-4f16-45e8-8f6d-8be062edf9f3" containerName="revoke-keys" Oct 30 21:58:19.218340 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:19.218317 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-bd3c59-dmzp9" Oct 30 21:58:19.274032 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:19.274006 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:58:19.277889 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:19.277859 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-bd3c59-dmzp9"] Oct 30 21:58:19.299890 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:19.299863 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/382cc232-70ae-4238-be44-1a7149ee4cbc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-bd3c59-dmzp9\" (UID: \"382cc232-70ae-4238-be44-1a7149ee4cbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-bd3c59-dmzp9" Oct 30 21:58:19.300026 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:19.299924 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp5hv\" (UniqueName: \"kubernetes.io/projected/382cc232-70ae-4238-be44-1a7149ee4cbc-kube-api-access-xp5hv\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-bd3c59-dmzp9\" (UID: \"382cc232-70ae-4238-be44-1a7149ee4cbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-bd3c59-dmzp9" Oct 30 21:58:19.300124 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:19.300098 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/382cc232-70ae-4238-be44-1a7149ee4cbc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-bd3c59-dmzp9\" (UID: \"382cc232-70ae-4238-be44-1a7149ee4cbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-bd3c59-dmzp9" Oct 30 21:58:19.324403 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:19.324374 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 21:58:19.324542 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:19.324476 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:58:19.324542 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:19.324507 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:58:19.401673 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:19.401635 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/382cc232-70ae-4238-be44-1a7149ee4cbc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-bd3c59-dmzp9\" (UID: \"382cc232-70ae-4238-be44-1a7149ee4cbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-bd3c59-dmzp9" Oct 30 21:58:19.401867 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:19.401770 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/382cc232-70ae-4238-be44-1a7149ee4cbc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-bd3c59-dmzp9\" (UID: \"382cc232-70ae-4238-be44-1a7149ee4cbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-bd3c59-dmzp9" Oct 30 21:58:19.401867 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:19.401816 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp5hv\" (UniqueName: \"kubernetes.io/projected/382cc232-70ae-4238-be44-1a7149ee4cbc-kube-api-access-xp5hv\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-bd3c59-dmzp9\" (UID: \"382cc232-70ae-4238-be44-1a7149ee4cbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-bd3c59-dmzp9" Oct 30 21:58:19.402397 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:19.402376 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/382cc232-70ae-4238-be44-1a7149ee4cbc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-bd3c59-dmzp9\" (UID: \"382cc232-70ae-4238-be44-1a7149ee4cbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-bd3c59-dmzp9" Oct 30 21:58:19.404243 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:19.404218 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/382cc232-70ae-4238-be44-1a7149ee4cbc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-bd3c59-dmzp9\" (UID: \"382cc232-70ae-4238-be44-1a7149ee4cbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-bd3c59-dmzp9" Oct 30 21:58:19.418842 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:19.418820 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29ee56e3-xrcd6"] Oct 30 21:58:19.422986 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:19.422965 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ee56e3-xrcd6" Oct 30 21:58:19.464520 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:19.464449 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp5hv\" (UniqueName: \"kubernetes.io/projected/382cc232-70ae-4238-be44-1a7149ee4cbc-kube-api-access-xp5hv\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-bd3c59-dmzp9\" (UID: \"382cc232-70ae-4238-be44-1a7149ee4cbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-bd3c59-dmzp9" Oct 30 21:58:19.487037 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:19.487005 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29ee56e3-xrcd6"] Oct 30 21:58:19.503306 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:19.503271 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4814d07d-359c-4dd4-97de-c575b2b5052f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29ee56e3-xrcd6\" (UID: \"4814d07d-359c-4dd4-97de-c575b2b5052f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ee56e3-xrcd6" Oct 30 21:58:19.503492 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:19.503330 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78dp4\" (UniqueName: \"kubernetes.io/projected/4814d07d-359c-4dd4-97de-c575b2b5052f-kube-api-access-78dp4\") pod \"maas-api-revoke-sub-29ee56e3-xrcd6\" (UID: \"4814d07d-359c-4dd4-97de-c575b2b5052f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ee56e3-xrcd6" Oct 30 21:58:19.503575 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:19.503548 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4814d07d-359c-4dd4-97de-c575b2b5052f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29ee56e3-xrcd6\" (UID: \"4814d07d-359c-4dd4-97de-c575b2b5052f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ee56e3-xrcd6" Oct 30 21:58:19.532344 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:19.532313 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-bd3c59-dmzp9" Oct 30 21:58:19.609649 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:19.609605 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4814d07d-359c-4dd4-97de-c575b2b5052f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29ee56e3-xrcd6\" (UID: \"4814d07d-359c-4dd4-97de-c575b2b5052f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ee56e3-xrcd6" Oct 30 21:58:19.609781 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:19.609675 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78dp4\" (UniqueName: \"kubernetes.io/projected/4814d07d-359c-4dd4-97de-c575b2b5052f-kube-api-access-78dp4\") pod \"maas-api-revoke-sub-29ee56e3-xrcd6\" (UID: \"4814d07d-359c-4dd4-97de-c575b2b5052f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ee56e3-xrcd6" Oct 30 21:58:19.609781 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:19.609751 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4814d07d-359c-4dd4-97de-c575b2b5052f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29ee56e3-xrcd6\" (UID: \"4814d07d-359c-4dd4-97de-c575b2b5052f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ee56e3-xrcd6" Oct 30 21:58:19.610758 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:19.610731 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4814d07d-359c-4dd4-97de-c575b2b5052f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29ee56e3-xrcd6\" (UID: \"4814d07d-359c-4dd4-97de-c575b2b5052f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ee56e3-xrcd6" Oct 30 21:58:19.618817 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:19.618779 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4814d07d-359c-4dd4-97de-c575b2b5052f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29ee56e3-xrcd6\" (UID: \"4814d07d-359c-4dd4-97de-c575b2b5052f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ee56e3-xrcd6" Oct 30 21:58:19.674503 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:19.674480 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78dp4\" (UniqueName: \"kubernetes.io/projected/4814d07d-359c-4dd4-97de-c575b2b5052f-kube-api-access-78dp4\") pod \"maas-api-revoke-sub-29ee56e3-xrcd6\" (UID: \"4814d07d-359c-4dd4-97de-c575b2b5052f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ee56e3-xrcd6" Oct 30 21:58:19.740279 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:19.740209 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ee56e3-xrcd6" Oct 30 21:58:19.920948 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:19.920921 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-bd3c59-dmzp9"] Oct 30 21:58:19.922708 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:58:19.922677 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod382cc232_70ae_4238_be44_1a7149ee4cbc.slice/crio-211030906f20ca9b9ab1f2cc2ba6d13eedfb52dfda0122696848abee0c5453b6 WatchSource:0}: Error finding container 211030906f20ca9b9ab1f2cc2ba6d13eedfb52dfda0122696848abee0c5453b6: Status 404 returned error can't find the container with id 211030906f20ca9b9ab1f2cc2ba6d13eedfb52dfda0122696848abee0c5453b6 Oct 30 21:58:20.142028 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:20.141995 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29ee56e3-xrcd6"] Oct 30 21:58:20.142590 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:58:20.142564 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4814d07d_359c_4dd4_97de_c575b2b5052f.slice/crio-6090a834b69abef93fda0644d700ea62bc4094bf54a404a0a42d0a07be129ce6 WatchSource:0}: Error finding container 6090a834b69abef93fda0644d700ea62bc4094bf54a404a0a42d0a07be129ce6: Status 404 returned error can't find the container with id 6090a834b69abef93fda0644d700ea62bc4094bf54a404a0a42d0a07be129ce6 Oct 30 21:58:20.232162 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:20.232133 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ee56e3-xrcd6" event={"ID":"4814d07d-359c-4dd4-97de-c575b2b5052f","Type":"ContainerStarted","Data":"6090a834b69abef93fda0644d700ea62bc4094bf54a404a0a42d0a07be129ce6"} Oct 30 21:58:20.233739 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:20.233713 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-bd3c59-dmzp9" event={"ID":"382cc232-70ae-4238-be44-1a7149ee4cbc","Type":"ContainerStarted","Data":"8631fd647f8cae67c2ffd457b79a187237484b2eaeef9122095c7b097b554a69"} Oct 30 21:58:20.233847 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:20.233758 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-bd3c59-dmzp9" event={"ID":"382cc232-70ae-4238-be44-1a7149ee4cbc","Type":"ContainerStarted","Data":"211030906f20ca9b9ab1f2cc2ba6d13eedfb52dfda0122696848abee0c5453b6"} Oct 30 21:58:20.354116 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:20.354069 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-bd3c59-dmzp9" podStartSLOduration=1.354054284 podStartE2EDuration="1.354054284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:20.351840568 +0000 UTC m=+2084.094361505" watchObservedRunningTime="2026-10-30 21:58:20.354054284 +0000 UTC m=+2084.096575545" Oct 30 21:58:21.242667 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:21.242623 5748 generic.go:356] "Generic (PLEG): container finished" podID="4814d07d-359c-4dd4-97de-c575b2b5052f" containerID="0b65111ae775092ffedb1c67b2d751164891386664fa1c78056c6ac7fe0a99ec" exitCode=0 Oct 30 21:58:21.243174 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:21.242698 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ee56e3-xrcd6" event={"ID":"4814d07d-359c-4dd4-97de-c575b2b5052f","Type":"ContainerDied","Data":"0b65111ae775092ffedb1c67b2d751164891386664fa1c78056c6ac7fe0a99ec"} Oct 30 21:58:21.244626 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:21.244606 5748 generic.go:356] "Generic (PLEG): container finished" podID="382cc232-70ae-4238-be44-1a7149ee4cbc" containerID="8631fd647f8cae67c2ffd457b79a187237484b2eaeef9122095c7b097b554a69" exitCode=0 Oct 30 21:58:21.244737 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:21.244642 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-bd3c59-dmzp9" event={"ID":"382cc232-70ae-4238-be44-1a7149ee4cbc","Type":"ContainerDied","Data":"8631fd647f8cae67c2ffd457b79a187237484b2eaeef9122095c7b097b554a69"} Oct 30 21:58:22.452853 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:22.452830 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-bd3c59-dmzp9" Oct 30 21:58:22.456203 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:22.456181 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ee56e3-xrcd6" Oct 30 21:58:22.547164 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:22.547085 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/382cc232-70ae-4238-be44-1a7149ee4cbc-kube-api-access-xp5hv\" (UniqueName: \"kubernetes.io/projected/382cc232-70ae-4238-be44-1a7149ee4cbc-kube-api-access-xp5hv\") pod \"382cc232-70ae-4238-be44-1a7149ee4cbc\" (UID: \"382cc232-70ae-4238-be44-1a7149ee4cbc\") " Oct 30 21:58:22.547164 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:22.547141 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/382cc232-70ae-4238-be44-1a7149ee4cbc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/382cc232-70ae-4238-be44-1a7149ee4cbc-maas-api-openshift-ca\") pod \"382cc232-70ae-4238-be44-1a7149ee4cbc\" (UID: \"382cc232-70ae-4238-be44-1a7149ee4cbc\") " Oct 30 21:58:22.547415 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:22.547233 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4814d07d-359c-4dd4-97de-c575b2b5052f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4814d07d-359c-4dd4-97de-c575b2b5052f-maas-api-xks-ca\") pod \"4814d07d-359c-4dd4-97de-c575b2b5052f\" (UID: \"4814d07d-359c-4dd4-97de-c575b2b5052f\") " Oct 30 21:58:22.547415 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:22.547298 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4814d07d-359c-4dd4-97de-c575b2b5052f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4814d07d-359c-4dd4-97de-c575b2b5052f-maas-api-openshift-ca\") pod \"4814d07d-359c-4dd4-97de-c575b2b5052f\" (UID: \"4814d07d-359c-4dd4-97de-c575b2b5052f\") " Oct 30 21:58:22.547415 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:22.547343 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/382cc232-70ae-4238-be44-1a7149ee4cbc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/382cc232-70ae-4238-be44-1a7149ee4cbc-maas-api-xks-ca\") pod \"382cc232-70ae-4238-be44-1a7149ee4cbc\" (UID: \"382cc232-70ae-4238-be44-1a7149ee4cbc\") " Oct 30 21:58:22.547415 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:22.547371 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4814d07d-359c-4dd4-97de-c575b2b5052f-kube-api-access-78dp4\" (UniqueName: \"kubernetes.io/projected/4814d07d-359c-4dd4-97de-c575b2b5052f-kube-api-access-78dp4\") pod \"4814d07d-359c-4dd4-97de-c575b2b5052f\" (UID: \"4814d07d-359c-4dd4-97de-c575b2b5052f\") " Oct 30 21:58:22.547694 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:22.547618 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/382cc232-70ae-4238-be44-1a7149ee4cbc-maas-api-openshift-ca" pod "382cc232-70ae-4238-be44-1a7149ee4cbc" (UID: "382cc232-70ae-4238-be44-1a7149ee4cbc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:58:22.547756 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:22.547710 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4814d07d-359c-4dd4-97de-c575b2b5052f-maas-api-openshift-ca" pod "4814d07d-359c-4dd4-97de-c575b2b5052f" (UID: "4814d07d-359c-4dd4-97de-c575b2b5052f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:58:22.548301 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:22.548213 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/382cc232-70ae-4238-be44-1a7149ee4cbc-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:22.548301 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:22.548268 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4814d07d-359c-4dd4-97de-c575b2b5052f-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:22.549630 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:22.549604 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/382cc232-70ae-4238-be44-1a7149ee4cbc-maas-api-xks-ca" pod "382cc232-70ae-4238-be44-1a7149ee4cbc" (UID: "382cc232-70ae-4238-be44-1a7149ee4cbc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:58:22.549630 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:22.549633 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/382cc232-70ae-4238-be44-1a7149ee4cbc-kube-api-access-xp5hv" pod "382cc232-70ae-4238-be44-1a7149ee4cbc" (UID: "382cc232-70ae-4238-be44-1a7149ee4cbc"). InnerVolumeSpecName "kube-api-access-xp5hv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:22.549781 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:22.549650 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4814d07d-359c-4dd4-97de-c575b2b5052f-maas-api-xks-ca" pod "4814d07d-359c-4dd4-97de-c575b2b5052f" (UID: "4814d07d-359c-4dd4-97de-c575b2b5052f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:58:22.549781 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:22.549679 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4814d07d-359c-4dd4-97de-c575b2b5052f-kube-api-access-78dp4" pod "4814d07d-359c-4dd4-97de-c575b2b5052f" (UID: "4814d07d-359c-4dd4-97de-c575b2b5052f"). InnerVolumeSpecName "kube-api-access-78dp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:22.649642 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:22.649607 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4814d07d-359c-4dd4-97de-c575b2b5052f-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:22.649642 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:22.649637 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/382cc232-70ae-4238-be44-1a7149ee4cbc-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:22.649642 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:22.649649 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78dp4\" (UniqueName: \"kubernetes.io/projected/4814d07d-359c-4dd4-97de-c575b2b5052f-kube-api-access-78dp4\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:22.649889 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:22.649660 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp5hv\" (UniqueName: \"kubernetes.io/projected/382cc232-70ae-4238-be44-1a7149ee4cbc-kube-api-access-xp5hv\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:23.261193 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:23.261162 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ee56e3-xrcd6" Oct 30 21:58:23.261490 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:23.261186 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ee56e3-xrcd6" event={"ID":"4814d07d-359c-4dd4-97de-c575b2b5052f","Type":"ContainerDied","Data":"6090a834b69abef93fda0644d700ea62bc4094bf54a404a0a42d0a07be129ce6"} Oct 30 21:58:23.261490 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:23.261253 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6090a834b69abef93fda0644d700ea62bc4094bf54a404a0a42d0a07be129ce6" Oct 30 21:58:23.263179 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:23.263155 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-bd3c59-dmzp9" Oct 30 21:58:23.263309 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:23.263189 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-bd3c59-dmzp9" event={"ID":"382cc232-70ae-4238-be44-1a7149ee4cbc","Type":"ContainerDied","Data":"211030906f20ca9b9ab1f2cc2ba6d13eedfb52dfda0122696848abee0c5453b6"} Oct 30 21:58:23.263309 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:23.263233 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="211030906f20ca9b9ab1f2cc2ba6d13eedfb52dfda0122696848abee0c5453b6" Oct 30 21:58:30.145837 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:58:30.145785 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:58:30.741038 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:30.740995 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-bd3c59-55b47bcd98-qtjwp"] Oct 30 21:58:34.982101 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:34.982066 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d6e0be-pmzv7"] Oct 30 21:58:35.035190 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:35.035152 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d6e0be-pmzv7"] Oct 30 21:58:35.742315 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:35.742278 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-bd3c59-55b47bcd98-qtjwp" podUID="fae90fa9-3d3c-4b91-ba40-b19bf6340d17" containerName="payload-processing" containerID="cri-o://9a68e2981606c5cae4e8571b3b577b87a73c1df1ad03cb75822974093c574138" gracePeriod=25 Oct 30 21:58:36.005809 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:36.005755 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-bd3c59-55b47bcd98-qtjwp" Oct 30 21:58:36.031601 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:36.031549 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fae90fa9-3d3c-4b91-ba40-b19bf6340d17-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fae90fa9-3d3c-4b91-ba40-b19bf6340d17-plugins-config-volume\") pod \"fae90fa9-3d3c-4b91-ba40-b19bf6340d17\" (UID: \"fae90fa9-3d3c-4b91-ba40-b19bf6340d17\") " Oct 30 21:58:36.031807 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:36.031634 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fae90fa9-3d3c-4b91-ba40-b19bf6340d17-kube-api-access-m2lsv\" (UniqueName: \"kubernetes.io/projected/fae90fa9-3d3c-4b91-ba40-b19bf6340d17-kube-api-access-m2lsv\") pod \"fae90fa9-3d3c-4b91-ba40-b19bf6340d17\" (UID: \"fae90fa9-3d3c-4b91-ba40-b19bf6340d17\") " Oct 30 21:58:36.032398 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:36.032353 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fae90fa9-3d3c-4b91-ba40-b19bf6340d17-plugins-config-volume" pod "fae90fa9-3d3c-4b91-ba40-b19bf6340d17" (UID: "fae90fa9-3d3c-4b91-ba40-b19bf6340d17"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:58:36.034561 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:36.034516 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fae90fa9-3d3c-4b91-ba40-b19bf6340d17-kube-api-access-m2lsv" pod "fae90fa9-3d3c-4b91-ba40-b19bf6340d17" (UID: "fae90fa9-3d3c-4b91-ba40-b19bf6340d17"). InnerVolumeSpecName "kube-api-access-m2lsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:36.133735 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:36.133699 5748 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fae90fa9-3d3c-4b91-ba40-b19bf6340d17-plugins-config-volume\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:36.133735 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:36.133731 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2lsv\" (UniqueName: \"kubernetes.io/projected/fae90fa9-3d3c-4b91-ba40-b19bf6340d17-kube-api-access-m2lsv\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:36.324673 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:36.324588 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e51c52-l8pv8"] Oct 30 21:58:36.327731 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:36.327706 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e51c52-l8pv8"] Oct 30 21:58:36.356266 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:36.356236 5748 generic.go:356] "Generic (PLEG): container finished" podID="fae90fa9-3d3c-4b91-ba40-b19bf6340d17" containerID="9a68e2981606c5cae4e8571b3b577b87a73c1df1ad03cb75822974093c574138" exitCode=0 Oct 30 21:58:36.356412 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:36.356298 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-bd3c59-55b47bcd98-qtjwp" event={"ID":"fae90fa9-3d3c-4b91-ba40-b19bf6340d17","Type":"ContainerDied","Data":"9a68e2981606c5cae4e8571b3b577b87a73c1df1ad03cb75822974093c574138"} Oct 30 21:58:36.356412 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:36.356305 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-bd3c59-55b47bcd98-qtjwp" Oct 30 21:58:36.356412 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:36.356360 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-bd3c59-55b47bcd98-qtjwp" event={"ID":"fae90fa9-3d3c-4b91-ba40-b19bf6340d17","Type":"ContainerDied","Data":"016701d0783cac5947586eebbbe6527d8fa9a857b925a1a3383c7c1d15727ff4"} Oct 30 21:58:36.356412 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:36.356382 5748 scope.go:122] "RemoveContainer" containerID="9a68e2981606c5cae4e8571b3b577b87a73c1df1ad03cb75822974093c574138" Oct 30 21:58:36.367343 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:36.367328 5748 scope.go:122] "RemoveContainer" containerID="9a68e2981606c5cae4e8571b3b577b87a73c1df1ad03cb75822974093c574138" Oct 30 21:58:36.367637 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:58:36.367617 5748 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a68e2981606c5cae4e8571b3b577b87a73c1df1ad03cb75822974093c574138\": container with ID starting with 9a68e2981606c5cae4e8571b3b577b87a73c1df1ad03cb75822974093c574138 not found: ID does not exist" containerID="9a68e2981606c5cae4e8571b3b577b87a73c1df1ad03cb75822974093c574138" Oct 30 21:58:36.367713 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:36.367646 5748 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a68e2981606c5cae4e8571b3b577b87a73c1df1ad03cb75822974093c574138"} err="failed to get container status \"9a68e2981606c5cae4e8571b3b577b87a73c1df1ad03cb75822974093c574138\": rpc error: code = NotFound desc = could not find container \"9a68e2981606c5cae4e8571b3b577b87a73c1df1ad03cb75822974093c574138\": container with ID starting with 9a68e2981606c5cae4e8571b3b577b87a73c1df1ad03cb75822974093c574138 not found: ID does not exist" Oct 30 21:58:36.416795 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:36.416767 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e3b179f-4f16-45e8-8f6d-8be062edf9f3" path="/var/lib/kubelet/pods/2e3b179f-4f16-45e8-8f6d-8be062edf9f3/volumes" Oct 30 21:58:36.417227 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:36.417212 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5da50f53-661c-4e80-b55b-4ae2ab155ceb" path="/var/lib/kubelet/pods/5da50f53-661c-4e80-b55b-4ae2ab155ceb/volumes" Oct 30 21:58:36.529673 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:36.529641 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-bd3c59-55b47bcd98-qtjwp"] Oct 30 21:58:36.535776 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:36.535749 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-bd3c59-55b47bcd98-qtjwp"] Oct 30 21:58:36.774743 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:36.774706 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j4v8r_3df036ff-1c00-4ce2-9e86-8acbeedf6c82/ovn-acl-logging/0.log" Oct 30 21:58:36.779923 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:36.779886 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j4v8r_3df036ff-1c00-4ce2-9e86-8acbeedf6c82/ovn-acl-logging/0.log" Oct 30 21:58:38.417992 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:38.417957 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fae90fa9-3d3c-4b91-ba40-b19bf6340d17" path="/var/lib/kubelet/pods/fae90fa9-3d3c-4b91-ba40-b19bf6340d17/volumes" Oct 30 21:58:42.996638 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:42.996608 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c92783-7b9gd"] Oct 30 21:58:43.046478 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:43.046443 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c92783-7b9gd"] Oct 30 21:58:44.025885 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:44.025851 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78c632-qnwqv"] Oct 30 21:58:44.026564 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:44.026543 5748 cpu_manager.go:414] "containerMap: removing container" podUID="382cc232-70ae-4238-be44-1a7149ee4cbc" containerName="revoke-keys" Oct 30 21:58:44.026564 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:44.026562 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="382cc232-70ae-4238-be44-1a7149ee4cbc" containerName="revoke-keys" Oct 30 21:58:44.026752 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:44.026577 5748 cpu_manager.go:414] "containerMap: removing container" podUID="fae90fa9-3d3c-4b91-ba40-b19bf6340d17" containerName="payload-processing" Oct 30 21:58:44.026752 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:44.026582 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fae90fa9-3d3c-4b91-ba40-b19bf6340d17" containerName="payload-processing" Oct 30 21:58:44.026752 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:44.026616 5748 cpu_manager.go:414] "containerMap: removing container" podUID="4814d07d-359c-4dd4-97de-c575b2b5052f" containerName="revoke-keys" Oct 30 21:58:44.026752 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:44.026621 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4814d07d-359c-4dd4-97de-c575b2b5052f" containerName="revoke-keys" Oct 30 21:58:44.026752 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:44.026741 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="4814d07d-359c-4dd4-97de-c575b2b5052f" containerName="revoke-keys" Oct 30 21:58:44.026752 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:44.026753 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="382cc232-70ae-4238-be44-1a7149ee4cbc" containerName="revoke-keys" Oct 30 21:58:44.027086 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:44.026760 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="fae90fa9-3d3c-4b91-ba40-b19bf6340d17" containerName="payload-processing" Oct 30 21:58:44.031284 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:44.031256 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78c632-qnwqv" Oct 30 21:58:44.083523 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:44.083480 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh"] Oct 30 21:58:44.084011 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:44.083828 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh" podUID="f3d0a7fa-e727-4776-b3c2-db1218760332" containerName="istio-proxy" containerID="cri-o://dc83b0f53a3fff3f61080c7c335b5bd0ef3a17e911eb6fa0cbd4d10350728fc4" gracePeriod=30 Oct 30 21:58:44.089415 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:44.089388 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:58:44.090462 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:44.090411 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78c632-qnwqv"] Oct 30 21:58:44.123777 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:44.123748 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:58:44.124018 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:44.123815 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 21:58:44.124314 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:44.123889 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:58:44.155719 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:44.155661 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mff5w\" (UniqueName: \"kubernetes.io/projected/370cd288-ede4-49c6-96f2-d7417b877384-kube-api-access-mff5w\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-78c632-qnwqv\" (UID: \"370cd288-ede4-49c6-96f2-d7417b877384\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78c632-qnwqv" Oct 30 21:58:44.155923 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:44.155781 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/370cd288-ede4-49c6-96f2-d7417b877384-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-78c632-qnwqv\" (UID: \"370cd288-ede4-49c6-96f2-d7417b877384\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78c632-qnwqv" Oct 30 21:58:44.156010 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:44.155953 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/370cd288-ede4-49c6-96f2-d7417b877384-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-78c632-qnwqv\" (UID: \"370cd288-ede4-49c6-96f2-d7417b877384\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78c632-qnwqv" Oct 30 21:58:44.257636 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:44.257596 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mff5w\" (UniqueName: \"kubernetes.io/projected/370cd288-ede4-49c6-96f2-d7417b877384-kube-api-access-mff5w\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-78c632-qnwqv\" (UID: \"370cd288-ede4-49c6-96f2-d7417b877384\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78c632-qnwqv" Oct 30 21:58:44.257818 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:44.257647 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/370cd288-ede4-49c6-96f2-d7417b877384-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-78c632-qnwqv\" (UID: \"370cd288-ede4-49c6-96f2-d7417b877384\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78c632-qnwqv" Oct 30 21:58:44.257818 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:44.257705 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/370cd288-ede4-49c6-96f2-d7417b877384-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-78c632-qnwqv\" (UID: \"370cd288-ede4-49c6-96f2-d7417b877384\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78c632-qnwqv" Oct 30 21:58:44.258441 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:44.258402 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/370cd288-ede4-49c6-96f2-d7417b877384-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-78c632-qnwqv\" (UID: \"370cd288-ede4-49c6-96f2-d7417b877384\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78c632-qnwqv" Oct 30 21:58:44.260132 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:44.260103 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/370cd288-ede4-49c6-96f2-d7417b877384-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-78c632-qnwqv\" (UID: \"370cd288-ede4-49c6-96f2-d7417b877384\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78c632-qnwqv" Oct 30 21:58:44.319460 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:44.319393 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mff5w\" (UniqueName: \"kubernetes.io/projected/370cd288-ede4-49c6-96f2-d7417b877384-kube-api-access-mff5w\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-78c632-qnwqv\" (UID: \"370cd288-ede4-49c6-96f2-d7417b877384\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78c632-qnwqv" Oct 30 21:58:44.348077 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:44.348050 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78c632-qnwqv" Oct 30 21:58:44.424504 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:44.424472 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15744eac-1e0a-49ba-892b-3425b609afc4" path="/var/lib/kubelet/pods/15744eac-1e0a-49ba-892b-3425b609afc4/volumes" Oct 30 21:58:44.725957 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:44.725929 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78c632-qnwqv"] Oct 30 21:58:44.727230 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:58:44.727197 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod370cd288_ede4_49c6_96f2_d7417b877384.slice/crio-efdc6a9a5cdcebd57c02a2dd81330555a64c7e30b1eae6c3a834ffd8af5b932d WatchSource:0}: Error finding container efdc6a9a5cdcebd57c02a2dd81330555a64c7e30b1eae6c3a834ffd8af5b932d: Status 404 returned error can't find the container with id efdc6a9a5cdcebd57c02a2dd81330555a64c7e30b1eae6c3a834ffd8af5b932d Oct 30 21:58:45.437581 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:45.437547 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78c632-qnwqv" event={"ID":"370cd288-ede4-49c6-96f2-d7417b877384","Type":"ContainerStarted","Data":"7af23afeef1c9d6eb1dea97dbe82176aa08b1586a6163c9d7c157a57ffa8168d"} Oct 30 21:58:45.437980 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:45.437602 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78c632-qnwqv" event={"ID":"370cd288-ede4-49c6-96f2-d7417b877384","Type":"ContainerStarted","Data":"efdc6a9a5cdcebd57c02a2dd81330555a64c7e30b1eae6c3a834ffd8af5b932d"} Oct 30 21:58:45.558359 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:45.558313 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78c632-qnwqv" podStartSLOduration=2.558299875 podStartE2EDuration="2.558299875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:45.555358112 +0000 UTC m=+2109.297879046" watchObservedRunningTime="2026-10-30 21:58:45.558299875 +0000 UTC m=+2109.300820810" Oct 30 21:58:46.445799 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:46.445760 5748 generic.go:356] "Generic (PLEG): container finished" podID="370cd288-ede4-49c6-96f2-d7417b877384" containerID="7af23afeef1c9d6eb1dea97dbe82176aa08b1586a6163c9d7c157a57ffa8168d" exitCode=0 Oct 30 21:58:46.446215 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:46.445841 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78c632-qnwqv" event={"ID":"370cd288-ede4-49c6-96f2-d7417b877384","Type":"ContainerDied","Data":"7af23afeef1c9d6eb1dea97dbe82176aa08b1586a6163c9d7c157a57ffa8168d"} Oct 30 21:58:47.624099 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:47.624064 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78c632-qnwqv" Oct 30 21:58:47.709154 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:47.709066 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/370cd288-ede4-49c6-96f2-d7417b877384-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/370cd288-ede4-49c6-96f2-d7417b877384-maas-api-xks-ca\") pod \"370cd288-ede4-49c6-96f2-d7417b877384\" (UID: \"370cd288-ede4-49c6-96f2-d7417b877384\") " Oct 30 21:58:47.709154 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:47.709130 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/370cd288-ede4-49c6-96f2-d7417b877384-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/370cd288-ede4-49c6-96f2-d7417b877384-maas-api-openshift-ca\") pod \"370cd288-ede4-49c6-96f2-d7417b877384\" (UID: \"370cd288-ede4-49c6-96f2-d7417b877384\") " Oct 30 21:58:47.709405 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:47.709249 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/370cd288-ede4-49c6-96f2-d7417b877384-kube-api-access-mff5w\" (UniqueName: \"kubernetes.io/projected/370cd288-ede4-49c6-96f2-d7417b877384-kube-api-access-mff5w\") pod \"370cd288-ede4-49c6-96f2-d7417b877384\" (UID: \"370cd288-ede4-49c6-96f2-d7417b877384\") " Oct 30 21:58:47.709523 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:47.709496 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/370cd288-ede4-49c6-96f2-d7417b877384-maas-api-openshift-ca" pod "370cd288-ede4-49c6-96f2-d7417b877384" (UID: "370cd288-ede4-49c6-96f2-d7417b877384"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:58:47.709794 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:47.709771 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/370cd288-ede4-49c6-96f2-d7417b877384-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:47.711390 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:47.711354 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/370cd288-ede4-49c6-96f2-d7417b877384-kube-api-access-mff5w" pod "370cd288-ede4-49c6-96f2-d7417b877384" (UID: "370cd288-ede4-49c6-96f2-d7417b877384"). InnerVolumeSpecName "kube-api-access-mff5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:47.711593 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:47.711574 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/370cd288-ede4-49c6-96f2-d7417b877384-maas-api-xks-ca" pod "370cd288-ede4-49c6-96f2-d7417b877384" (UID: "370cd288-ede4-49c6-96f2-d7417b877384"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:58:47.811100 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:47.811058 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/370cd288-ede4-49c6-96f2-d7417b877384-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:47.811100 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:47.811094 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mff5w\" (UniqueName: \"kubernetes.io/projected/370cd288-ede4-49c6-96f2-d7417b877384-kube-api-access-mff5w\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:48.461789 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:48.461741 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78c632-qnwqv" event={"ID":"370cd288-ede4-49c6-96f2-d7417b877384","Type":"ContainerDied","Data":"efdc6a9a5cdcebd57c02a2dd81330555a64c7e30b1eae6c3a834ffd8af5b932d"} Oct 30 21:58:48.462025 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:48.461814 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="efdc6a9a5cdcebd57c02a2dd81330555a64c7e30b1eae6c3a834ffd8af5b932d" Oct 30 21:58:48.462025 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:48.461753 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78c632-qnwqv" Oct 30 21:58:49.382981 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:49.382953 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh" Oct 30 21:58:49.432057 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:49.432025 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f3d0a7fa-e727-4776-b3c2-db1218760332-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3d0a7fa-e727-4776-b3c2-db1218760332-istio-podinfo\") pod \"f3d0a7fa-e727-4776-b3c2-db1218760332\" (UID: \"f3d0a7fa-e727-4776-b3c2-db1218760332\") " Oct 30 21:58:49.432320 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:49.432119 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3d0a7fa-e727-4776-b3c2-db1218760332-istio-token\" (UniqueName: \"kubernetes.io/projected/f3d0a7fa-e727-4776-b3c2-db1218760332-istio-token\") pod \"f3d0a7fa-e727-4776-b3c2-db1218760332\" (UID: \"f3d0a7fa-e727-4776-b3c2-db1218760332\") " Oct 30 21:58:49.432320 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:49.432214 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3d0a7fa-e727-4776-b3c2-db1218760332-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3d0a7fa-e727-4776-b3c2-db1218760332-istio-data\") pod \"f3d0a7fa-e727-4776-b3c2-db1218760332\" (UID: \"f3d0a7fa-e727-4776-b3c2-db1218760332\") " Oct 30 21:58:49.432320 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:49.432281 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3d0a7fa-e727-4776-b3c2-db1218760332-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3d0a7fa-e727-4776-b3c2-db1218760332-workload-certs\") pod \"f3d0a7fa-e727-4776-b3c2-db1218760332\" (UID: \"f3d0a7fa-e727-4776-b3c2-db1218760332\") " Oct 30 21:58:49.432320 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:49.432308 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3d0a7fa-e727-4776-b3c2-db1218760332-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d0a7fa-e727-4776-b3c2-db1218760332-workload-socket\") pod \"f3d0a7fa-e727-4776-b3c2-db1218760332\" (UID: \"f3d0a7fa-e727-4776-b3c2-db1218760332\") " Oct 30 21:58:49.432589 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:49.432407 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3d0a7fa-e727-4776-b3c2-db1218760332-kube-api-access-5klvr\" (UniqueName: \"kubernetes.io/projected/f3d0a7fa-e727-4776-b3c2-db1218760332-kube-api-access-5klvr\") pod \"f3d0a7fa-e727-4776-b3c2-db1218760332\" (UID: \"f3d0a7fa-e727-4776-b3c2-db1218760332\") " Oct 30 21:58:49.432589 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:49.432441 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3d0a7fa-e727-4776-b3c2-db1218760332-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d0a7fa-e727-4776-b3c2-db1218760332-credential-socket\") pod \"f3d0a7fa-e727-4776-b3c2-db1218760332\" (UID: \"f3d0a7fa-e727-4776-b3c2-db1218760332\") " Oct 30 21:58:49.432589 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:49.432505 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3d0a7fa-e727-4776-b3c2-db1218760332-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3d0a7fa-e727-4776-b3c2-db1218760332-istio-envoy\") pod \"f3d0a7fa-e727-4776-b3c2-db1218760332\" (UID: \"f3d0a7fa-e727-4776-b3c2-db1218760332\") " Oct 30 21:58:49.432589 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:49.432563 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3d0a7fa-e727-4776-b3c2-db1218760332-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3d0a7fa-e727-4776-b3c2-db1218760332-istiod-ca-cert\") pod \"f3d0a7fa-e727-4776-b3c2-db1218760332\" (UID: \"f3d0a7fa-e727-4776-b3c2-db1218760332\") " Oct 30 21:58:49.433274 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:49.433026 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3d0a7fa-e727-4776-b3c2-db1218760332-workload-socket" pod "f3d0a7fa-e727-4776-b3c2-db1218760332" (UID: "f3d0a7fa-e727-4776-b3c2-db1218760332"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:49.433274 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:49.433223 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3d0a7fa-e727-4776-b3c2-db1218760332-workload-certs" pod "f3d0a7fa-e727-4776-b3c2-db1218760332" (UID: "f3d0a7fa-e727-4776-b3c2-db1218760332"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:49.433274 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:49.433240 5748 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d0a7fa-e727-4776-b3c2-db1218760332-workload-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:49.433510 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:49.433319 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3d0a7fa-e727-4776-b3c2-db1218760332-istiod-ca-cert" pod "f3d0a7fa-e727-4776-b3c2-db1218760332" (UID: "f3d0a7fa-e727-4776-b3c2-db1218760332"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:58:49.433813 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:49.433788 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3d0a7fa-e727-4776-b3c2-db1218760332-istio-data" pod "f3d0a7fa-e727-4776-b3c2-db1218760332" (UID: "f3d0a7fa-e727-4776-b3c2-db1218760332"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:49.434033 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:49.434002 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3d0a7fa-e727-4776-b3c2-db1218760332-credential-socket" pod "f3d0a7fa-e727-4776-b3c2-db1218760332" (UID: "f3d0a7fa-e727-4776-b3c2-db1218760332"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:49.435550 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:49.435519 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3d0a7fa-e727-4776-b3c2-db1218760332-istio-token" pod "f3d0a7fa-e727-4776-b3c2-db1218760332" (UID: "f3d0a7fa-e727-4776-b3c2-db1218760332"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:49.436197 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:49.436175 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3d0a7fa-e727-4776-b3c2-db1218760332-istio-envoy" pod "f3d0a7fa-e727-4776-b3c2-db1218760332" (UID: "f3d0a7fa-e727-4776-b3c2-db1218760332"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:49.436519 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:49.436490 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3d0a7fa-e727-4776-b3c2-db1218760332-kube-api-access-5klvr" pod "f3d0a7fa-e727-4776-b3c2-db1218760332" (UID: "f3d0a7fa-e727-4776-b3c2-db1218760332"). InnerVolumeSpecName "kube-api-access-5klvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:49.437043 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:49.437020 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3d0a7fa-e727-4776-b3c2-db1218760332-istio-podinfo" pod "f3d0a7fa-e727-4776-b3c2-db1218760332" (UID: "f3d0a7fa-e727-4776-b3c2-db1218760332"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:58:49.472201 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:49.472167 5748 generic.go:356] "Generic (PLEG): container finished" podID="f3d0a7fa-e727-4776-b3c2-db1218760332" containerID="dc83b0f53a3fff3f61080c7c335b5bd0ef3a17e911eb6fa0cbd4d10350728fc4" exitCode=0 Oct 30 21:58:49.472363 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:49.472223 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh" Oct 30 21:58:49.472363 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:49.472251 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh" event={"ID":"f3d0a7fa-e727-4776-b3c2-db1218760332","Type":"ContainerDied","Data":"dc83b0f53a3fff3f61080c7c335b5bd0ef3a17e911eb6fa0cbd4d10350728fc4"} Oct 30 21:58:49.472363 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:49.472308 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh" event={"ID":"f3d0a7fa-e727-4776-b3c2-db1218760332","Type":"ContainerDied","Data":"d32bfb19244f10ea6c5d4f03015f28cf84e4b91f4408c35beb3e302de40c7c99"} Oct 30 21:58:49.472363 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:49.472332 5748 scope.go:122] "RemoveContainer" containerID="dc83b0f53a3fff3f61080c7c335b5bd0ef3a17e911eb6fa0cbd4d10350728fc4" Oct 30 21:58:49.483165 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:49.483149 5748 scope.go:122] "RemoveContainer" containerID="eaa55b38e0c967494780c31fd90097d9a1a29c32f3bfc7f22c3245291dbda6ca" Oct 30 21:58:49.492350 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:49.492330 5748 scope.go:122] "RemoveContainer" containerID="dc83b0f53a3fff3f61080c7c335b5bd0ef3a17e911eb6fa0cbd4d10350728fc4" Oct 30 21:58:49.492623 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:58:49.492604 5748 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc83b0f53a3fff3f61080c7c335b5bd0ef3a17e911eb6fa0cbd4d10350728fc4\": container with ID starting with dc83b0f53a3fff3f61080c7c335b5bd0ef3a17e911eb6fa0cbd4d10350728fc4 not found: ID does not exist" containerID="dc83b0f53a3fff3f61080c7c335b5bd0ef3a17e911eb6fa0cbd4d10350728fc4" Oct 30 21:58:49.492681 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:49.492630 5748 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc83b0f53a3fff3f61080c7c335b5bd0ef3a17e911eb6fa0cbd4d10350728fc4"} err="failed to get container status \"dc83b0f53a3fff3f61080c7c335b5bd0ef3a17e911eb6fa0cbd4d10350728fc4\": rpc error: code = NotFound desc = could not find container \"dc83b0f53a3fff3f61080c7c335b5bd0ef3a17e911eb6fa0cbd4d10350728fc4\": container with ID starting with dc83b0f53a3fff3f61080c7c335b5bd0ef3a17e911eb6fa0cbd4d10350728fc4 not found: ID does not exist" Oct 30 21:58:49.492681 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:49.492648 5748 scope.go:122] "RemoveContainer" containerID="eaa55b38e0c967494780c31fd90097d9a1a29c32f3bfc7f22c3245291dbda6ca" Oct 30 21:58:49.492877 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:58:49.492855 5748 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaa55b38e0c967494780c31fd90097d9a1a29c32f3bfc7f22c3245291dbda6ca\": container with ID starting with eaa55b38e0c967494780c31fd90097d9a1a29c32f3bfc7f22c3245291dbda6ca not found: ID does not exist" containerID="eaa55b38e0c967494780c31fd90097d9a1a29c32f3bfc7f22c3245291dbda6ca" Oct 30 21:58:49.492959 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:49.492885 5748 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaa55b38e0c967494780c31fd90097d9a1a29c32f3bfc7f22c3245291dbda6ca"} err="failed to get container status \"eaa55b38e0c967494780c31fd90097d9a1a29c32f3bfc7f22c3245291dbda6ca\": rpc error: code = NotFound desc = could not find container \"eaa55b38e0c967494780c31fd90097d9a1a29c32f3bfc7f22c3245291dbda6ca\": container with ID starting with eaa55b38e0c967494780c31fd90097d9a1a29c32f3bfc7f22c3245291dbda6ca not found: ID does not exist" Oct 30 21:58:49.534872 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:49.534847 5748 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3d0a7fa-e727-4776-b3c2-db1218760332-istio-data\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:49.534872 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:49.534872 5748 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3d0a7fa-e727-4776-b3c2-db1218760332-workload-certs\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:49.535044 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:49.534884 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5klvr\" (UniqueName: \"kubernetes.io/projected/f3d0a7fa-e727-4776-b3c2-db1218760332-kube-api-access-5klvr\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:49.535044 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:49.534923 5748 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d0a7fa-e727-4776-b3c2-db1218760332-credential-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:49.535044 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:49.534934 5748 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3d0a7fa-e727-4776-b3c2-db1218760332-istio-envoy\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:49.535044 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:49.534948 5748 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3d0a7fa-e727-4776-b3c2-db1218760332-istiod-ca-cert\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:49.535044 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:49.534958 5748 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3d0a7fa-e727-4776-b3c2-db1218760332-istio-podinfo\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:49.535044 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:49.534966 5748 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3d0a7fa-e727-4776-b3c2-db1218760332-istio-token\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:49.671616 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:49.671583 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh"] Oct 30 21:58:49.680138 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:49.680106 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-e51c52-openshift-default-86c9f5569f-z4rnh"] Oct 30 21:58:50.418669 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:50.418631 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3d0a7fa-e727-4776-b3c2-db1218760332" path="/var/lib/kubelet/pods/f3d0a7fa-e727-4776-b3c2-db1218760332/volumes" Oct 30 21:58:55.667423 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:58:55.667327 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-78c632-7784567c9b-bm6kt"] Oct 30 21:59:00.147847 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:59:00.147806 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:59:00.668143 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:00.668106 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-78c632-7784567c9b-bm6kt" podUID="950134bb-c2f9-4de9-a6a2-366a8e30f1b0" containerName="payload-processing" containerID="cri-o://5381fb88994223f2889d6f45110f7668f5dc4000340e90cf34763599f4e3f2ee" gracePeriod=25 Oct 30 21:59:00.930619 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:00.930563 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-78c632-7784567c9b-bm6kt" Oct 30 21:59:01.000317 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:01.000293 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/950134bb-c2f9-4de9-a6a2-366a8e30f1b0-kube-api-access-fgrpr\" (UniqueName: \"kubernetes.io/projected/950134bb-c2f9-4de9-a6a2-366a8e30f1b0-kube-api-access-fgrpr\") pod \"950134bb-c2f9-4de9-a6a2-366a8e30f1b0\" (UID: \"950134bb-c2f9-4de9-a6a2-366a8e30f1b0\") " Oct 30 21:59:01.000453 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:01.000329 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/950134bb-c2f9-4de9-a6a2-366a8e30f1b0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/950134bb-c2f9-4de9-a6a2-366a8e30f1b0-plugins-config-volume\") pod \"950134bb-c2f9-4de9-a6a2-366a8e30f1b0\" (UID: \"950134bb-c2f9-4de9-a6a2-366a8e30f1b0\") " Oct 30 21:59:01.000714 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:01.000687 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/950134bb-c2f9-4de9-a6a2-366a8e30f1b0-plugins-config-volume" pod "950134bb-c2f9-4de9-a6a2-366a8e30f1b0" (UID: "950134bb-c2f9-4de9-a6a2-366a8e30f1b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:01.000967 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:01.000950 5748 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/950134bb-c2f9-4de9-a6a2-366a8e30f1b0-plugins-config-volume\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:01.002401 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:01.002375 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/950134bb-c2f9-4de9-a6a2-366a8e30f1b0-kube-api-access-fgrpr" pod "950134bb-c2f9-4de9-a6a2-366a8e30f1b0" (UID: "950134bb-c2f9-4de9-a6a2-366a8e30f1b0"). InnerVolumeSpecName "kube-api-access-fgrpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:01.102693 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:01.102662 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgrpr\" (UniqueName: \"kubernetes.io/projected/950134bb-c2f9-4de9-a6a2-366a8e30f1b0-kube-api-access-fgrpr\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:01.561329 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:01.561297 5748 generic.go:356] "Generic (PLEG): container finished" podID="950134bb-c2f9-4de9-a6a2-366a8e30f1b0" containerID="5381fb88994223f2889d6f45110f7668f5dc4000340e90cf34763599f4e3f2ee" exitCode=0 Oct 30 21:59:01.561329 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:01.561336 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-78c632-7784567c9b-bm6kt" Oct 30 21:59:01.561803 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:01.561370 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-78c632-7784567c9b-bm6kt" event={"ID":"950134bb-c2f9-4de9-a6a2-366a8e30f1b0","Type":"ContainerDied","Data":"5381fb88994223f2889d6f45110f7668f5dc4000340e90cf34763599f4e3f2ee"} Oct 30 21:59:01.561803 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:01.561428 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-78c632-7784567c9b-bm6kt" event={"ID":"950134bb-c2f9-4de9-a6a2-366a8e30f1b0","Type":"ContainerDied","Data":"20bbb7b6f9bb1dce9249d1f77e544dff5bce96ff48ae743417c3aaf73054136c"} Oct 30 21:59:01.561803 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:01.561456 5748 scope.go:122] "RemoveContainer" containerID="5381fb88994223f2889d6f45110f7668f5dc4000340e90cf34763599f4e3f2ee" Oct 30 21:59:01.572229 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:01.572208 5748 scope.go:122] "RemoveContainer" containerID="5381fb88994223f2889d6f45110f7668f5dc4000340e90cf34763599f4e3f2ee" Oct 30 21:59:01.572542 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:59:01.572516 5748 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5381fb88994223f2889d6f45110f7668f5dc4000340e90cf34763599f4e3f2ee\": container with ID starting with 5381fb88994223f2889d6f45110f7668f5dc4000340e90cf34763599f4e3f2ee not found: ID does not exist" containerID="5381fb88994223f2889d6f45110f7668f5dc4000340e90cf34763599f4e3f2ee" Oct 30 21:59:01.572613 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:01.572549 5748 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5381fb88994223f2889d6f45110f7668f5dc4000340e90cf34763599f4e3f2ee"} err="failed to get container status \"5381fb88994223f2889d6f45110f7668f5dc4000340e90cf34763599f4e3f2ee\": rpc error: code = NotFound desc = could not find container \"5381fb88994223f2889d6f45110f7668f5dc4000340e90cf34763599f4e3f2ee\": container with ID starting with 5381fb88994223f2889d6f45110f7668f5dc4000340e90cf34763599f4e3f2ee not found: ID does not exist" Oct 30 21:59:01.740840 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:01.740807 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-78c632-7784567c9b-bm6kt"] Oct 30 21:59:01.743416 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:01.743389 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5bfa09-7nkhz"] Oct 30 21:59:01.744198 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:01.744160 5748 cpu_manager.go:414] "containerMap: removing container" podUID="f3d0a7fa-e727-4776-b3c2-db1218760332" containerName="istio-proxy" Oct 30 21:59:01.744278 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:01.744201 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3d0a7fa-e727-4776-b3c2-db1218760332" containerName="istio-proxy" Oct 30 21:59:01.744278 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:01.744216 5748 cpu_manager.go:414] "containerMap: removing container" podUID="950134bb-c2f9-4de9-a6a2-366a8e30f1b0" containerName="payload-processing" Oct 30 21:59:01.744278 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:01.744224 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="950134bb-c2f9-4de9-a6a2-366a8e30f1b0" containerName="payload-processing" Oct 30 21:59:01.744431 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:01.744283 5748 cpu_manager.go:414] "containerMap: removing container" podUID="370cd288-ede4-49c6-96f2-d7417b877384" containerName="revoke-keys" Oct 30 21:59:01.744431 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:01.744291 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="370cd288-ede4-49c6-96f2-d7417b877384" containerName="revoke-keys" Oct 30 21:59:01.744533 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:01.744431 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3d0a7fa-e727-4776-b3c2-db1218760332" containerName="istio-proxy" Oct 30 21:59:01.744533 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:01.744442 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="370cd288-ede4-49c6-96f2-d7417b877384" containerName="revoke-keys" Oct 30 21:59:01.744533 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:01.744457 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3d0a7fa-e727-4776-b3c2-db1218760332" containerName="istio-proxy" Oct 30 21:59:01.744533 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:01.744466 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="950134bb-c2f9-4de9-a6a2-366a8e30f1b0" containerName="payload-processing" Oct 30 21:59:01.749394 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:01.749373 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-78c632-7784567c9b-bm6kt"] Oct 30 21:59:01.749513 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:01.749469 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5bfa09-7nkhz" Oct 30 21:59:01.802873 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:01.802832 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:59:01.803155 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:01.803135 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 21:59:01.803229 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:01.803203 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5bfa09-7nkhz"] Oct 30 21:59:01.808306 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:01.808288 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:59:01.808306 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:01.808294 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:59:01.811076 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:01.811051 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50af2e13-b025-4186-a762-b2fa7350b2cc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-5bfa09-7nkhz\" (UID: \"50af2e13-b025-4186-a762-b2fa7350b2cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5bfa09-7nkhz" Oct 30 21:59:01.811196 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:01.811096 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcjxk\" (UniqueName: \"kubernetes.io/projected/50af2e13-b025-4186-a762-b2fa7350b2cc-kube-api-access-qcjxk\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-5bfa09-7nkhz\" (UID: \"50af2e13-b025-4186-a762-b2fa7350b2cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5bfa09-7nkhz" Oct 30 21:59:01.811259 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:01.811234 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50af2e13-b025-4186-a762-b2fa7350b2cc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-5bfa09-7nkhz\" (UID: \"50af2e13-b025-4186-a762-b2fa7350b2cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5bfa09-7nkhz" Oct 30 21:59:01.912836 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:01.912743 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50af2e13-b025-4186-a762-b2fa7350b2cc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-5bfa09-7nkhz\" (UID: \"50af2e13-b025-4186-a762-b2fa7350b2cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5bfa09-7nkhz" Oct 30 21:59:01.912836 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:01.912803 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcjxk\" (UniqueName: \"kubernetes.io/projected/50af2e13-b025-4186-a762-b2fa7350b2cc-kube-api-access-qcjxk\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-5bfa09-7nkhz\" (UID: \"50af2e13-b025-4186-a762-b2fa7350b2cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5bfa09-7nkhz" Oct 30 21:59:01.913098 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:01.912948 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50af2e13-b025-4186-a762-b2fa7350b2cc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-5bfa09-7nkhz\" (UID: \"50af2e13-b025-4186-a762-b2fa7350b2cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5bfa09-7nkhz" Oct 30 21:59:01.913568 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:01.913548 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50af2e13-b025-4186-a762-b2fa7350b2cc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-5bfa09-7nkhz\" (UID: \"50af2e13-b025-4186-a762-b2fa7350b2cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5bfa09-7nkhz" Oct 30 21:59:01.915290 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:01.915263 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50af2e13-b025-4186-a762-b2fa7350b2cc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-5bfa09-7nkhz\" (UID: \"50af2e13-b025-4186-a762-b2fa7350b2cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5bfa09-7nkhz" Oct 30 21:59:01.974512 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:01.974476 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcjxk\" (UniqueName: \"kubernetes.io/projected/50af2e13-b025-4186-a762-b2fa7350b2cc-kube-api-access-qcjxk\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-5bfa09-7nkhz\" (UID: \"50af2e13-b025-4186-a762-b2fa7350b2cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5bfa09-7nkhz" Oct 30 21:59:02.066803 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:02.066773 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5bfa09-7nkhz" Oct 30 21:59:02.418037 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:02.418008 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="950134bb-c2f9-4de9-a6a2-366a8e30f1b0" path="/var/lib/kubelet/pods/950134bb-c2f9-4de9-a6a2-366a8e30f1b0/volumes" Oct 30 21:59:02.450885 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:02.450857 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5bfa09-7nkhz"] Oct 30 21:59:02.452398 ip-10-0-143-61 kubenswrapper[5748]: W1030 21:59:02.452372 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50af2e13_b025_4186_a762_b2fa7350b2cc.slice/crio-be6cfc040284cc2ac380a0198784e381b3735ca361c80b87d0f6b0f901010069 WatchSource:0}: Error finding container be6cfc040284cc2ac380a0198784e381b3735ca361c80b87d0f6b0f901010069: Status 404 returned error can't find the container with id be6cfc040284cc2ac380a0198784e381b3735ca361c80b87d0f6b0f901010069 Oct 30 21:59:02.569031 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:02.568991 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5bfa09-7nkhz" event={"ID":"50af2e13-b025-4186-a762-b2fa7350b2cc","Type":"ContainerStarted","Data":"be6cfc040284cc2ac380a0198784e381b3735ca361c80b87d0f6b0f901010069"} Oct 30 21:59:03.580040 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:03.579964 5748 generic.go:356] "Generic (PLEG): container finished" podID="50af2e13-b025-4186-a762-b2fa7350b2cc" containerID="4e1d1dbc36a8b295e0a60dacf00b47d78272698d2cfbf8f13b7fffc22061883a" exitCode=0 Oct 30 21:59:03.580040 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:03.580011 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5bfa09-7nkhz" event={"ID":"50af2e13-b025-4186-a762-b2fa7350b2cc","Type":"ContainerDied","Data":"4e1d1dbc36a8b295e0a60dacf00b47d78272698d2cfbf8f13b7fffc22061883a"} Oct 30 21:59:04.753118 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:04.753090 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5bfa09-7nkhz" Oct 30 21:59:04.855078 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:04.854994 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50af2e13-b025-4186-a762-b2fa7350b2cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50af2e13-b025-4186-a762-b2fa7350b2cc-maas-api-openshift-ca\") pod \"50af2e13-b025-4186-a762-b2fa7350b2cc\" (UID: \"50af2e13-b025-4186-a762-b2fa7350b2cc\") " Oct 30 21:59:04.855078 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:04.855040 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50af2e13-b025-4186-a762-b2fa7350b2cc-kube-api-access-qcjxk\" (UniqueName: \"kubernetes.io/projected/50af2e13-b025-4186-a762-b2fa7350b2cc-kube-api-access-qcjxk\") pod \"50af2e13-b025-4186-a762-b2fa7350b2cc\" (UID: \"50af2e13-b025-4186-a762-b2fa7350b2cc\") " Oct 30 21:59:04.855331 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:04.855180 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50af2e13-b025-4186-a762-b2fa7350b2cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50af2e13-b025-4186-a762-b2fa7350b2cc-maas-api-xks-ca\") pod \"50af2e13-b025-4186-a762-b2fa7350b2cc\" (UID: \"50af2e13-b025-4186-a762-b2fa7350b2cc\") " Oct 30 21:59:04.855418 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:04.855382 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50af2e13-b025-4186-a762-b2fa7350b2cc-maas-api-openshift-ca" pod "50af2e13-b025-4186-a762-b2fa7350b2cc" (UID: "50af2e13-b025-4186-a762-b2fa7350b2cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:04.855750 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:04.855727 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50af2e13-b025-4186-a762-b2fa7350b2cc-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:04.857722 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:04.857698 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50af2e13-b025-4186-a762-b2fa7350b2cc-maas-api-xks-ca" pod "50af2e13-b025-4186-a762-b2fa7350b2cc" (UID: "50af2e13-b025-4186-a762-b2fa7350b2cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:59:04.857846 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:04.857750 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50af2e13-b025-4186-a762-b2fa7350b2cc-kube-api-access-qcjxk" pod "50af2e13-b025-4186-a762-b2fa7350b2cc" (UID: "50af2e13-b025-4186-a762-b2fa7350b2cc"). InnerVolumeSpecName "kube-api-access-qcjxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:04.957542 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:04.957508 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50af2e13-b025-4186-a762-b2fa7350b2cc-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:04.957542 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:04.957541 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcjxk\" (UniqueName: \"kubernetes.io/projected/50af2e13-b025-4186-a762-b2fa7350b2cc-kube-api-access-qcjxk\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:05.595126 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:05.595088 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5bfa09-7nkhz" Oct 30 21:59:05.595126 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:05.595105 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5bfa09-7nkhz" event={"ID":"50af2e13-b025-4186-a762-b2fa7350b2cc","Type":"ContainerDied","Data":"be6cfc040284cc2ac380a0198784e381b3735ca361c80b87d0f6b0f901010069"} Oct 30 21:59:05.595340 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:05.595163 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be6cfc040284cc2ac380a0198784e381b3735ca361c80b87d0f6b0f901010069" Oct 30 21:59:27.587388 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:27.587352 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-bd3c59-dmzp9"] Oct 30 21:59:27.590536 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:27.590509 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-bd3c59-dmzp9"] Oct 30 21:59:28.417358 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:28.417326 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="382cc232-70ae-4238-be44-1a7149ee4cbc" path="/var/lib/kubelet/pods/382cc232-70ae-4238-be44-1a7149ee4cbc/volumes" Oct 30 21:59:30.146313 ip-10-0-143-61 kubenswrapper[5748]: E1030 21:59:30.146276 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 21:59:58.272126 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:58.272093 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5bfa09-7nkhz"] Oct 30 21:59:58.321843 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:58.321812 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5bfa09-7nkhz"] Oct 30 21:59:58.417984 ip-10-0-143-61 kubenswrapper[5748]: I1030 21:59:58.417954 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50af2e13-b025-4186-a762-b2fa7350b2cc" path="/var/lib/kubelet/pods/50af2e13-b025-4186-a762-b2fa7350b2cc/volumes" Oct 30 22:00:00.151284 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:00:00.151244 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:00:00.173985 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.173956 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c7d477-29889960-l5cjm"] Oct 30 22:00:00.174585 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.174566 5748 cpu_manager.go:414] "containerMap: removing container" podUID="50af2e13-b025-4186-a762-b2fa7350b2cc" containerName="revoke-keys" Oct 30 22:00:00.174585 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.174584 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50af2e13-b025-4186-a762-b2fa7350b2cc" containerName="revoke-keys" Oct 30 22:00:00.174801 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.174622 5748 cpu_manager.go:414] "containerMap: removing container" podUID="f3d0a7fa-e727-4776-b3c2-db1218760332" containerName="istio-proxy" Oct 30 22:00:00.174801 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.174630 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3d0a7fa-e727-4776-b3c2-db1218760332" containerName="istio-proxy" Oct 30 22:00:00.174801 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.174747 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="50af2e13-b025-4186-a762-b2fa7350b2cc" containerName="revoke-keys" Oct 30 22:00:00.178191 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.178171 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889960-hm42n"] Oct 30 22:00:00.178326 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.178286 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c7d477-29889960-l5cjm" Oct 30 22:00:00.182104 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.182081 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-71a3d788-29889960-t7cvk"] Oct 30 22:00:00.182232 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.182212 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889960-hm42n" Oct 30 22:00:00.183754 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.183726 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67f2eccf-1409-448c-b69d-eb739ab2195e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-c7d477-29889960-l5cjm\" (UID: \"67f2eccf-1409-448c-b69d-eb739ab2195e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c7d477-29889960-l5cjm" Oct 30 22:00:00.183925 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.183854 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qpgx\" (UniqueName: \"kubernetes.io/projected/67f2eccf-1409-448c-b69d-eb739ab2195e-kube-api-access-7qpgx\") pod \"maas-api-key-cleanup-e2e-worker-w5-c7d477-29889960-l5cjm\" (UID: \"67f2eccf-1409-448c-b69d-eb739ab2195e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c7d477-29889960-l5cjm" Oct 30 22:00:00.185731 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.185712 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-71a3d788-29889960-t7cvk" Oct 30 22:00:00.224137 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.224101 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ea374-29889960-68g75"] Oct 30 22:00:00.228113 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.228092 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91874e-29889960-nj2rv"] Oct 30 22:00:00.228253 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.228220 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ea374-29889960-68g75" Oct 30 22:00:00.231782 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.231752 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:00:00.231889 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.231871 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2c8f56-29889960-m5257"] Oct 30 22:00:00.232091 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.232072 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91874e-29889960-nj2rv" Oct 30 22:00:00.235536 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.235502 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1de75a-29889960-rl48f"] Oct 30 22:00:00.235660 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.235590 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2c8f56-29889960-m5257" Oct 30 22:00:00.239578 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.239561 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1de75a-29889960-rl48f" Oct 30 22:00:00.271701 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.271674 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:00:00.271701 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.271677 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 22:00:00.274650 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.274623 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c7d477-29889960-l5cjm"] Oct 30 22:00:00.277240 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.277219 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889960-hm42n"] Oct 30 22:00:00.279395 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.279375 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91874e-29889960-nj2rv"] Oct 30 22:00:00.281444 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.281424 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ea374-29889960-68g75"] Oct 30 22:00:00.285077 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.285057 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qpgx\" (UniqueName: \"kubernetes.io/projected/67f2eccf-1409-448c-b69d-eb739ab2195e-kube-api-access-7qpgx\") pod \"maas-api-key-cleanup-e2e-worker-w5-c7d477-29889960-l5cjm\" (UID: \"67f2eccf-1409-448c-b69d-eb739ab2195e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c7d477-29889960-l5cjm" Oct 30 22:00:00.285151 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.285095 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f46b5473-f092-4bbb-8486-3c2b1da66231-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-1de75a-29889960-rl48f\" (UID: \"f46b5473-f092-4bbb-8486-3c2b1da66231\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1de75a-29889960-rl48f" Oct 30 22:00:00.285252 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.285227 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9a3ac51-bd06-4343-b765-5d60748fb62d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889960-hm42n\" (UID: \"a9a3ac51-bd06-4343-b765-5d60748fb62d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889960-hm42n" Oct 30 22:00:00.285335 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.285312 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wqwp\" (UniqueName: \"kubernetes.io/projected/da3ff95c-c45b-4f37-a080-9ed58a709e1d-kube-api-access-9wqwp\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-2c8f56-29889960-m5257\" (UID: \"da3ff95c-c45b-4f37-a080-9ed58a709e1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2c8f56-29889960-m5257" Oct 30 22:00:00.285418 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.285400 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13a18f96-f1ff-436a-8cb9-78d656a45ee4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-6ea374-29889960-68g75\" (UID: \"13a18f96-f1ff-436a-8cb9-78d656a45ee4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ea374-29889960-68g75" Oct 30 22:00:00.285510 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.285489 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbm2k\" (UniqueName: \"kubernetes.io/projected/6990f1c8-d49d-4151-a79a-e231ca0986a4-kube-api-access-tbm2k\") pod \"maas-api-key-cleanup-e2e-worker-w2-91874e-29889960-nj2rv\" (UID: \"6990f1c8-d49d-4151-a79a-e231ca0986a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91874e-29889960-nj2rv" Oct 30 22:00:00.285644 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.285516 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j226\" (UniqueName: \"kubernetes.io/projected/a9a3ac51-bd06-4343-b765-5d60748fb62d-kube-api-access-7j226\") pod \"maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889960-hm42n\" (UID: \"a9a3ac51-bd06-4343-b765-5d60748fb62d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889960-hm42n" Oct 30 22:00:00.285644 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.285535 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da3ff95c-c45b-4f37-a080-9ed58a709e1d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-2c8f56-29889960-m5257\" (UID: \"da3ff95c-c45b-4f37-a080-9ed58a709e1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2c8f56-29889960-m5257" Oct 30 22:00:00.285644 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.285624 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr8hk\" (UniqueName: \"kubernetes.io/projected/13a18f96-f1ff-436a-8cb9-78d656a45ee4-kube-api-access-pr8hk\") pod \"maas-api-key-cleanup-e2e-worker-w1-6ea374-29889960-68g75\" (UID: \"13a18f96-f1ff-436a-8cb9-78d656a45ee4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ea374-29889960-68g75" Oct 30 22:00:00.285949 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.285881 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67f2eccf-1409-448c-b69d-eb739ab2195e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-c7d477-29889960-l5cjm\" (UID: \"67f2eccf-1409-448c-b69d-eb739ab2195e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c7d477-29889960-l5cjm" Oct 30 22:00:00.286118 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.285965 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvf58\" (UniqueName: \"kubernetes.io/projected/3be41157-90f9-4290-b4e3-9db0bf8f5d9c-kube-api-access-bvf58\") pod \"maas-api-key-cleanup-e2e-mt-71a3d788-29889960-t7cvk\" (UID: \"3be41157-90f9-4290-b4e3-9db0bf8f5d9c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-71a3d788-29889960-t7cvk" Oct 30 22:00:00.286118 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.286042 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz9rm\" (UniqueName: \"kubernetes.io/projected/f46b5473-f092-4bbb-8486-3c2b1da66231-kube-api-access-qz9rm\") pod \"maas-api-key-cleanup-e2e-worker-w4-1de75a-29889960-rl48f\" (UID: \"f46b5473-f092-4bbb-8486-3c2b1da66231\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1de75a-29889960-rl48f" Oct 30 22:00:00.286118 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.286071 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6990f1c8-d49d-4151-a79a-e231ca0986a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-91874e-29889960-nj2rv\" (UID: \"6990f1c8-d49d-4151-a79a-e231ca0986a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91874e-29889960-nj2rv" Oct 30 22:00:00.286276 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.286161 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3be41157-90f9-4290-b4e3-9db0bf8f5d9c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-71a3d788-29889960-t7cvk\" (UID: \"3be41157-90f9-4290-b4e3-9db0bf8f5d9c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-71a3d788-29889960-t7cvk" Oct 30 22:00:00.286793 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.286773 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67f2eccf-1409-448c-b69d-eb739ab2195e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-c7d477-29889960-l5cjm\" (UID: \"67f2eccf-1409-448c-b69d-eb739ab2195e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c7d477-29889960-l5cjm" Oct 30 22:00:00.329911 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.327833 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2c8f56-29889960-m5257"] Oct 30 22:00:00.331438 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.331202 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1de75a-29889960-rl48f"] Oct 30 22:00:00.349086 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.349053 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qpgx\" (UniqueName: \"kubernetes.io/projected/67f2eccf-1409-448c-b69d-eb739ab2195e-kube-api-access-7qpgx\") pod \"maas-api-key-cleanup-e2e-worker-w5-c7d477-29889960-l5cjm\" (UID: \"67f2eccf-1409-448c-b69d-eb739ab2195e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c7d477-29889960-l5cjm" Oct 30 22:00:00.357754 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.357729 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-71a3d788-29889960-t7cvk"] Oct 30 22:00:00.387501 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.387464 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr8hk\" (UniqueName: \"kubernetes.io/projected/13a18f96-f1ff-436a-8cb9-78d656a45ee4-kube-api-access-pr8hk\") pod \"maas-api-key-cleanup-e2e-worker-w1-6ea374-29889960-68g75\" (UID: \"13a18f96-f1ff-436a-8cb9-78d656a45ee4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ea374-29889960-68g75" Oct 30 22:00:00.387662 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.387506 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvf58\" (UniqueName: \"kubernetes.io/projected/3be41157-90f9-4290-b4e3-9db0bf8f5d9c-kube-api-access-bvf58\") pod \"maas-api-key-cleanup-e2e-mt-71a3d788-29889960-t7cvk\" (UID: \"3be41157-90f9-4290-b4e3-9db0bf8f5d9c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-71a3d788-29889960-t7cvk" Oct 30 22:00:00.387662 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.387540 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz9rm\" (UniqueName: \"kubernetes.io/projected/f46b5473-f092-4bbb-8486-3c2b1da66231-kube-api-access-qz9rm\") pod \"maas-api-key-cleanup-e2e-worker-w4-1de75a-29889960-rl48f\" (UID: \"f46b5473-f092-4bbb-8486-3c2b1da66231\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1de75a-29889960-rl48f" Oct 30 22:00:00.387662 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.387564 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6990f1c8-d49d-4151-a79a-e231ca0986a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-91874e-29889960-nj2rv\" (UID: \"6990f1c8-d49d-4151-a79a-e231ca0986a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91874e-29889960-nj2rv" Oct 30 22:00:00.387662 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.387610 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3be41157-90f9-4290-b4e3-9db0bf8f5d9c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-71a3d788-29889960-t7cvk\" (UID: \"3be41157-90f9-4290-b4e3-9db0bf8f5d9c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-71a3d788-29889960-t7cvk" Oct 30 22:00:00.387892 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.387692 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f46b5473-f092-4bbb-8486-3c2b1da66231-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-1de75a-29889960-rl48f\" (UID: \"f46b5473-f092-4bbb-8486-3c2b1da66231\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1de75a-29889960-rl48f" Oct 30 22:00:00.387892 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.387749 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9a3ac51-bd06-4343-b765-5d60748fb62d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889960-hm42n\" (UID: \"a9a3ac51-bd06-4343-b765-5d60748fb62d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889960-hm42n" Oct 30 22:00:00.387892 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.387782 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wqwp\" (UniqueName: \"kubernetes.io/projected/da3ff95c-c45b-4f37-a080-9ed58a709e1d-kube-api-access-9wqwp\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-2c8f56-29889960-m5257\" (UID: \"da3ff95c-c45b-4f37-a080-9ed58a709e1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2c8f56-29889960-m5257" Oct 30 22:00:00.387892 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.387834 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13a18f96-f1ff-436a-8cb9-78d656a45ee4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-6ea374-29889960-68g75\" (UID: \"13a18f96-f1ff-436a-8cb9-78d656a45ee4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ea374-29889960-68g75" Oct 30 22:00:00.388138 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.387960 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbm2k\" (UniqueName: \"kubernetes.io/projected/6990f1c8-d49d-4151-a79a-e231ca0986a4-kube-api-access-tbm2k\") pod \"maas-api-key-cleanup-e2e-worker-w2-91874e-29889960-nj2rv\" (UID: \"6990f1c8-d49d-4151-a79a-e231ca0986a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91874e-29889960-nj2rv" Oct 30 22:00:00.388138 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.387997 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j226\" (UniqueName: \"kubernetes.io/projected/a9a3ac51-bd06-4343-b765-5d60748fb62d-kube-api-access-7j226\") pod \"maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889960-hm42n\" (UID: \"a9a3ac51-bd06-4343-b765-5d60748fb62d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889960-hm42n" Oct 30 22:00:00.388138 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.388031 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da3ff95c-c45b-4f37-a080-9ed58a709e1d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-2c8f56-29889960-m5257\" (UID: \"da3ff95c-c45b-4f37-a080-9ed58a709e1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2c8f56-29889960-m5257" Oct 30 22:00:00.388410 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.388386 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f46b5473-f092-4bbb-8486-3c2b1da66231-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-1de75a-29889960-rl48f\" (UID: \"f46b5473-f092-4bbb-8486-3c2b1da66231\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1de75a-29889960-rl48f" Oct 30 22:00:00.388511 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.388488 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6990f1c8-d49d-4151-a79a-e231ca0986a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-91874e-29889960-nj2rv\" (UID: \"6990f1c8-d49d-4151-a79a-e231ca0986a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91874e-29889960-nj2rv" Oct 30 22:00:00.388579 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.388492 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3be41157-90f9-4290-b4e3-9db0bf8f5d9c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-71a3d788-29889960-t7cvk\" (UID: \"3be41157-90f9-4290-b4e3-9db0bf8f5d9c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-71a3d788-29889960-t7cvk" Oct 30 22:00:00.388785 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.388762 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da3ff95c-c45b-4f37-a080-9ed58a709e1d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-2c8f56-29889960-m5257\" (UID: \"da3ff95c-c45b-4f37-a080-9ed58a709e1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2c8f56-29889960-m5257" Oct 30 22:00:00.389095 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.389072 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13a18f96-f1ff-436a-8cb9-78d656a45ee4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-6ea374-29889960-68g75\" (UID: \"13a18f96-f1ff-436a-8cb9-78d656a45ee4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ea374-29889960-68g75" Oct 30 22:00:00.389200 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.389182 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9a3ac51-bd06-4343-b765-5d60748fb62d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889960-hm42n\" (UID: \"a9a3ac51-bd06-4343-b765-5d60748fb62d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889960-hm42n" Oct 30 22:00:00.450610 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.450500 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbm2k\" (UniqueName: \"kubernetes.io/projected/6990f1c8-d49d-4151-a79a-e231ca0986a4-kube-api-access-tbm2k\") pod \"maas-api-key-cleanup-e2e-worker-w2-91874e-29889960-nj2rv\" (UID: \"6990f1c8-d49d-4151-a79a-e231ca0986a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91874e-29889960-nj2rv" Oct 30 22:00:00.450792 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.450775 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvf58\" (UniqueName: \"kubernetes.io/projected/3be41157-90f9-4290-b4e3-9db0bf8f5d9c-kube-api-access-bvf58\") pod \"maas-api-key-cleanup-e2e-mt-71a3d788-29889960-t7cvk\" (UID: \"3be41157-90f9-4290-b4e3-9db0bf8f5d9c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-71a3d788-29889960-t7cvk" Oct 30 22:00:00.450934 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.450879 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j226\" (UniqueName: \"kubernetes.io/projected/a9a3ac51-bd06-4343-b765-5d60748fb62d-kube-api-access-7j226\") pod \"maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889960-hm42n\" (UID: \"a9a3ac51-bd06-4343-b765-5d60748fb62d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889960-hm42n" Oct 30 22:00:00.451120 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.451094 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wqwp\" (UniqueName: \"kubernetes.io/projected/da3ff95c-c45b-4f37-a080-9ed58a709e1d-kube-api-access-9wqwp\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-2c8f56-29889960-m5257\" (UID: \"da3ff95c-c45b-4f37-a080-9ed58a709e1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2c8f56-29889960-m5257" Oct 30 22:00:00.451166 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.451116 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr8hk\" (UniqueName: \"kubernetes.io/projected/13a18f96-f1ff-436a-8cb9-78d656a45ee4-kube-api-access-pr8hk\") pod \"maas-api-key-cleanup-e2e-worker-w1-6ea374-29889960-68g75\" (UID: \"13a18f96-f1ff-436a-8cb9-78d656a45ee4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ea374-29889960-68g75" Oct 30 22:00:00.451274 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.451259 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz9rm\" (UniqueName: \"kubernetes.io/projected/f46b5473-f092-4bbb-8486-3c2b1da66231-kube-api-access-qz9rm\") pod \"maas-api-key-cleanup-e2e-worker-w4-1de75a-29889960-rl48f\" (UID: \"f46b5473-f092-4bbb-8486-3c2b1da66231\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1de75a-29889960-rl48f" Oct 30 22:00:00.495499 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.495460 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c7d477-29889960-l5cjm" Oct 30 22:00:00.502903 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.502871 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889960-hm42n" Oct 30 22:00:00.509812 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.509789 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-71a3d788-29889960-t7cvk" Oct 30 22:00:00.544166 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.544021 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ea374-29889960-68g75" Oct 30 22:00:00.555460 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.554137 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91874e-29889960-nj2rv" Oct 30 22:00:00.560198 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.560177 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2c8f56-29889960-m5257" Oct 30 22:00:00.576386 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.576169 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1de75a-29889960-rl48f" Oct 30 22:00:00.881281 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.881251 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c7d477-29889960-l5cjm"] Oct 30 22:00:00.881784 ip-10-0-143-61 kubenswrapper[5748]: W1030 22:00:00.881756 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67f2eccf_1409_448c_b69d_eb739ab2195e.slice/crio-9729627ede3439fcf0f2ae04f5fad89bdb5adc3f4385b4392584baa10c08975e WatchSource:0}: Error finding container 9729627ede3439fcf0f2ae04f5fad89bdb5adc3f4385b4392584baa10c08975e: Status 404 returned error can't find the container with id 9729627ede3439fcf0f2ae04f5fad89bdb5adc3f4385b4392584baa10c08975e Oct 30 22:00:00.928206 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.928139 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-71a3d788-29889960-t7cvk"] Oct 30 22:00:00.928642 ip-10-0-143-61 kubenswrapper[5748]: W1030 22:00:00.928614 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3be41157_90f9_4290_b4e3_9db0bf8f5d9c.slice/crio-28a83fca76b32658256da0a92b0775578db21704fb78fadaf7294c792a074ee2 WatchSource:0}: Error finding container 28a83fca76b32658256da0a92b0775578db21704fb78fadaf7294c792a074ee2: Status 404 returned error can't find the container with id 28a83fca76b32658256da0a92b0775578db21704fb78fadaf7294c792a074ee2 Oct 30 22:00:00.930482 ip-10-0-143-61 kubenswrapper[5748]: W1030 22:00:00.930455 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9a3ac51_bd06_4343_b765_5d60748fb62d.slice/crio-c759d015017e47c69c805a7ebda118ff0fb94c76045e97beb6acb50654f9ea5b WatchSource:0}: Error finding container c759d015017e47c69c805a7ebda118ff0fb94c76045e97beb6acb50654f9ea5b: Status 404 returned error can't find the container with id c759d015017e47c69c805a7ebda118ff0fb94c76045e97beb6acb50654f9ea5b Oct 30 22:00:00.931098 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.931081 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889960-hm42n"] Oct 30 22:00:00.961960 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.961926 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889960-hm42n" event={"ID":"a9a3ac51-bd06-4343-b765-5d60748fb62d","Type":"ContainerStarted","Data":"c759d015017e47c69c805a7ebda118ff0fb94c76045e97beb6acb50654f9ea5b"} Oct 30 22:00:00.963412 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.963388 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c7d477-29889960-l5cjm" event={"ID":"67f2eccf-1409-448c-b69d-eb739ab2195e","Type":"ContainerStarted","Data":"9729627ede3439fcf0f2ae04f5fad89bdb5adc3f4385b4392584baa10c08975e"} Oct 30 22:00:00.964510 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.964492 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-71a3d788-29889960-t7cvk" event={"ID":"3be41157-90f9-4290-b4e3-9db0bf8f5d9c","Type":"ContainerStarted","Data":"28a83fca76b32658256da0a92b0775578db21704fb78fadaf7294c792a074ee2"} Oct 30 22:00:00.982296 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:00.982267 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ea374-29889960-68g75"] Oct 30 22:00:00.984098 ip-10-0-143-61 kubenswrapper[5748]: W1030 22:00:00.984064 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13a18f96_f1ff_436a_8cb9_78d656a45ee4.slice/crio-33d00890901c2643f7d3ac6acbe621553b75e7932c2cee410dafea28a4f0968a WatchSource:0}: Error finding container 33d00890901c2643f7d3ac6acbe621553b75e7932c2cee410dafea28a4f0968a: Status 404 returned error can't find the container with id 33d00890901c2643f7d3ac6acbe621553b75e7932c2cee410dafea28a4f0968a Oct 30 22:00:01.002717 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:01.002688 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2c8f56-29889960-m5257"] Oct 30 22:00:01.003775 ip-10-0-143-61 kubenswrapper[5748]: W1030 22:00:01.003748 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda3ff95c_c45b_4f37_a080_9ed58a709e1d.slice/crio-e786c9085af175bff687f5810fe50ead2555508f74723882025792b687677c67 WatchSource:0}: Error finding container e786c9085af175bff687f5810fe50ead2555508f74723882025792b687677c67: Status 404 returned error can't find the container with id e786c9085af175bff687f5810fe50ead2555508f74723882025792b687677c67 Oct 30 22:00:01.010093 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:01.010058 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1de75a-29889960-rl48f"] Oct 30 22:00:01.010539 ip-10-0-143-61 kubenswrapper[5748]: W1030 22:00:01.010520 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf46b5473_f092_4bbb_8486_3c2b1da66231.slice/crio-3830c9f34046cba61c63619f5b8537caa1e0557d33b288dd38b4374bdfa4740a WatchSource:0}: Error finding container 3830c9f34046cba61c63619f5b8537caa1e0557d33b288dd38b4374bdfa4740a: Status 404 returned error can't find the container with id 3830c9f34046cba61c63619f5b8537caa1e0557d33b288dd38b4374bdfa4740a Oct 30 22:00:01.026526 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:01.026505 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91874e-29889960-nj2rv"] Oct 30 22:00:01.028860 ip-10-0-143-61 kubenswrapper[5748]: W1030 22:00:01.028834 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6990f1c8_d49d_4151_a79a_e231ca0986a4.slice/crio-77d4e5f0b9aa586f2ab22f5d33ca39560bf659051e9b71265761784618d7e6ac WatchSource:0}: Error finding container 77d4e5f0b9aa586f2ab22f5d33ca39560bf659051e9b71265761784618d7e6ac: Status 404 returned error can't find the container with id 77d4e5f0b9aa586f2ab22f5d33ca39560bf659051e9b71265761784618d7e6ac Oct 30 22:00:01.972710 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:01.972681 5748 generic.go:356] "Generic (PLEG): container finished" podID="a9a3ac51-bd06-4343-b765-5d60748fb62d" containerID="bf347107617b8330363a16b3f849ef787de26c1c04f8d38bd05b85fdc9481289" exitCode=60 Oct 30 22:00:01.973113 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:01.972759 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889960-hm42n" event={"ID":"a9a3ac51-bd06-4343-b765-5d60748fb62d","Type":"ContainerDied","Data":"bf347107617b8330363a16b3f849ef787de26c1c04f8d38bd05b85fdc9481289"} Oct 30 22:00:01.973113 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:01.973088 5748 scope.go:122] "RemoveContainer" containerID="bf347107617b8330363a16b3f849ef787de26c1c04f8d38bd05b85fdc9481289" Oct 30 22:00:01.974604 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:01.974579 5748 generic.go:356] "Generic (PLEG): container finished" podID="67f2eccf-1409-448c-b69d-eb739ab2195e" containerID="c369db16dffe45378d37da632b68b858148316ea42ab3aac5449f39a1fe07510" exitCode=60 Oct 30 22:00:01.974714 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:01.974644 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c7d477-29889960-l5cjm" event={"ID":"67f2eccf-1409-448c-b69d-eb739ab2195e","Type":"ContainerDied","Data":"c369db16dffe45378d37da632b68b858148316ea42ab3aac5449f39a1fe07510"} Oct 30 22:00:01.974932 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:01.974909 5748 scope.go:122] "RemoveContainer" containerID="c369db16dffe45378d37da632b68b858148316ea42ab3aac5449f39a1fe07510" Oct 30 22:00:01.976466 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:01.976444 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1de75a-29889960-rl48f" event={"ID":"f46b5473-f092-4bbb-8486-3c2b1da66231","Type":"ContainerStarted","Data":"5d2652fdf5d7718453b85451cc4e19c21dc13472215f8a648729fab60ef9fbb3"} Oct 30 22:00:01.976523 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:01.976492 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1de75a-29889960-rl48f" event={"ID":"f46b5473-f092-4bbb-8486-3c2b1da66231","Type":"ContainerStarted","Data":"3830c9f34046cba61c63619f5b8537caa1e0557d33b288dd38b4374bdfa4740a"} Oct 30 22:00:01.978239 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:01.978214 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ea374-29889960-68g75" event={"ID":"13a18f96-f1ff-436a-8cb9-78d656a45ee4","Type":"ContainerStarted","Data":"1bfb723049732074b9e2ee80fd2b96c5998ca99f749e6fe8a1819f463f766c22"} Oct 30 22:00:01.978315 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:01.978262 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ea374-29889960-68g75" event={"ID":"13a18f96-f1ff-436a-8cb9-78d656a45ee4","Type":"ContainerStarted","Data":"33d00890901c2643f7d3ac6acbe621553b75e7932c2cee410dafea28a4f0968a"} Oct 30 22:00:01.979945 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:01.979923 5748 generic.go:356] "Generic (PLEG): container finished" podID="3be41157-90f9-4290-b4e3-9db0bf8f5d9c" containerID="2aa50ea0a4681e07033546044fcc6d217180b2c499f81e0339c0d32c21305550" exitCode=60 Oct 30 22:00:01.980054 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:01.979946 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-71a3d788-29889960-t7cvk" event={"ID":"3be41157-90f9-4290-b4e3-9db0bf8f5d9c","Type":"ContainerDied","Data":"2aa50ea0a4681e07033546044fcc6d217180b2c499f81e0339c0d32c21305550"} Oct 30 22:00:01.980199 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:01.980183 5748 scope.go:122] "RemoveContainer" containerID="2aa50ea0a4681e07033546044fcc6d217180b2c499f81e0339c0d32c21305550" Oct 30 22:00:01.981539 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:01.981515 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91874e-29889960-nj2rv" event={"ID":"6990f1c8-d49d-4151-a79a-e231ca0986a4","Type":"ContainerStarted","Data":"784ed1aac7174d304e4198986737aa48393b67ee7ba3b879ddd94863577db777"} Oct 30 22:00:01.981630 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:01.981559 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91874e-29889960-nj2rv" event={"ID":"6990f1c8-d49d-4151-a79a-e231ca0986a4","Type":"ContainerStarted","Data":"77d4e5f0b9aa586f2ab22f5d33ca39560bf659051e9b71265761784618d7e6ac"} Oct 30 22:00:01.983111 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:01.983087 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2c8f56-29889960-m5257" event={"ID":"da3ff95c-c45b-4f37-a080-9ed58a709e1d","Type":"ContainerStarted","Data":"d01f8c8127952e95ccfe1d55ddc33eb10bbedfdf297f87b6a31ccd8060c82a58"} Oct 30 22:00:01.983211 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:01.983134 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2c8f56-29889960-m5257" event={"ID":"da3ff95c-c45b-4f37-a080-9ed58a709e1d","Type":"ContainerStarted","Data":"e786c9085af175bff687f5810fe50ead2555508f74723882025792b687677c67"} Oct 30 22:00:02.324458 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:02.324403 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91874e-29889960-nj2rv" podStartSLOduration=2.324389292 podStartE2EDuration="2.324389292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:02.322586911 +0000 UTC m=+2186.065107840" watchObservedRunningTime="2026-10-30 22:00:02.324389292 +0000 UTC m=+2186.066910230" Oct 30 22:00:02.438835 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:02.438783 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ea374-29889960-68g75" podStartSLOduration=2.438770021 podStartE2EDuration="2.438770021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:02.438390134 +0000 UTC m=+2186.180911069" watchObservedRunningTime="2026-10-30 22:00:02.438770021 +0000 UTC m=+2186.181290956" Oct 30 22:00:02.556982 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:02.556929 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1de75a-29889960-rl48f" podStartSLOduration=2.556915225 podStartE2EDuration="2.556915225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:02.554740648 +0000 UTC m=+2186.297261582" watchObservedRunningTime="2026-10-30 22:00:02.556915225 +0000 UTC m=+2186.299436157" Oct 30 22:00:02.671408 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:02.671353 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2c8f56-29889960-m5257" podStartSLOduration=2.671337613 podStartE2EDuration="2.671337613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:02.670948897 +0000 UTC m=+2186.413469833" watchObservedRunningTime="2026-10-30 22:00:02.671337613 +0000 UTC m=+2186.413858549" Oct 30 22:00:02.992275 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:02.992241 5748 generic.go:356] "Generic (PLEG): container finished" podID="6990f1c8-d49d-4151-a79a-e231ca0986a4" containerID="784ed1aac7174d304e4198986737aa48393b67ee7ba3b879ddd94863577db777" exitCode=60 Oct 30 22:00:02.992725 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:02.992329 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91874e-29889960-nj2rv" event={"ID":"6990f1c8-d49d-4151-a79a-e231ca0986a4","Type":"ContainerDied","Data":"784ed1aac7174d304e4198986737aa48393b67ee7ba3b879ddd94863577db777"} Oct 30 22:00:02.992725 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:02.992593 5748 scope.go:122] "RemoveContainer" containerID="784ed1aac7174d304e4198986737aa48393b67ee7ba3b879ddd94863577db777" Oct 30 22:00:02.994037 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:02.994014 5748 generic.go:356] "Generic (PLEG): container finished" podID="da3ff95c-c45b-4f37-a080-9ed58a709e1d" containerID="d01f8c8127952e95ccfe1d55ddc33eb10bbedfdf297f87b6a31ccd8060c82a58" exitCode=60 Oct 30 22:00:02.994159 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:02.994096 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2c8f56-29889960-m5257" event={"ID":"da3ff95c-c45b-4f37-a080-9ed58a709e1d","Type":"ContainerDied","Data":"d01f8c8127952e95ccfe1d55ddc33eb10bbedfdf297f87b6a31ccd8060c82a58"} Oct 30 22:00:02.994322 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:02.994308 5748 scope.go:122] "RemoveContainer" containerID="d01f8c8127952e95ccfe1d55ddc33eb10bbedfdf297f87b6a31ccd8060c82a58" Oct 30 22:00:02.996134 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:02.996112 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889960-hm42n" event={"ID":"a9a3ac51-bd06-4343-b765-5d60748fb62d","Type":"ContainerStarted","Data":"6e8e272f8c1e055abd8c2a799fbcabfc30cfb900a4e51373678b1f2a2c2e503b"} Oct 30 22:00:02.998355 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:02.998332 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c7d477-29889960-l5cjm" event={"ID":"67f2eccf-1409-448c-b69d-eb739ab2195e","Type":"ContainerStarted","Data":"4690451fb7c69388388edc0497f29a1947d814a398d25a062b4e3cb330d39d77"} Oct 30 22:00:03.000004 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:02.999977 5748 generic.go:356] "Generic (PLEG): container finished" podID="f46b5473-f092-4bbb-8486-3c2b1da66231" containerID="5d2652fdf5d7718453b85451cc4e19c21dc13472215f8a648729fab60ef9fbb3" exitCode=60 Oct 30 22:00:03.000097 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:03.000052 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1de75a-29889960-rl48f" event={"ID":"f46b5473-f092-4bbb-8486-3c2b1da66231","Type":"ContainerDied","Data":"5d2652fdf5d7718453b85451cc4e19c21dc13472215f8a648729fab60ef9fbb3"} Oct 30 22:00:03.000305 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:03.000288 5748 scope.go:122] "RemoveContainer" containerID="5d2652fdf5d7718453b85451cc4e19c21dc13472215f8a648729fab60ef9fbb3" Oct 30 22:00:03.001828 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:03.001805 5748 generic.go:356] "Generic (PLEG): container finished" podID="13a18f96-f1ff-436a-8cb9-78d656a45ee4" containerID="1bfb723049732074b9e2ee80fd2b96c5998ca99f749e6fe8a1819f463f766c22" exitCode=60 Oct 30 22:00:03.001969 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:03.001880 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ea374-29889960-68g75" event={"ID":"13a18f96-f1ff-436a-8cb9-78d656a45ee4","Type":"ContainerDied","Data":"1bfb723049732074b9e2ee80fd2b96c5998ca99f749e6fe8a1819f463f766c22"} Oct 30 22:00:03.002103 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:03.002086 5748 scope.go:122] "RemoveContainer" containerID="1bfb723049732074b9e2ee80fd2b96c5998ca99f749e6fe8a1819f463f766c22" Oct 30 22:00:03.004371 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:03.004349 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-71a3d788-29889960-t7cvk" event={"ID":"3be41157-90f9-4290-b4e3-9db0bf8f5d9c","Type":"ContainerStarted","Data":"6ad30539cd9fc545425e990bc81341e1099e04f3b0dbc4182683f837938c3dbc"} Oct 30 22:00:03.345314 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:03.345266 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c7d477-29889960-l5cjm" podStartSLOduration=3.345251364 podStartE2EDuration="3.345251364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:03.225732162 +0000 UTC m=+2186.968253097" watchObservedRunningTime="2026-10-30 22:00:03.345251364 +0000 UTC m=+2187.087772298" Oct 30 22:00:03.577515 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:03.577379 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889960-hm42n" podStartSLOduration=3.577362043 podStartE2EDuration="3.577362043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:03.577224589 +0000 UTC m=+2187.319745526" watchObservedRunningTime="2026-10-30 22:00:03.577362043 +0000 UTC m=+2187.319882979" Oct 30 22:00:03.818143 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:03.818096 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-71a3d788-29889960-t7cvk" podStartSLOduration=3.818083416 podStartE2EDuration="3.818083416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:03.815604482 +0000 UTC m=+2187.558125428" watchObservedRunningTime="2026-10-30 22:00:03.818083416 +0000 UTC m=+2187.560604405" Oct 30 22:00:04.013316 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:04.013280 5748 generic.go:356] "Generic (PLEG): container finished" podID="3be41157-90f9-4290-b4e3-9db0bf8f5d9c" containerID="6ad30539cd9fc545425e990bc81341e1099e04f3b0dbc4182683f837938c3dbc" exitCode=60 Oct 30 22:00:04.013762 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:04.013358 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-71a3d788-29889960-t7cvk" event={"ID":"3be41157-90f9-4290-b4e3-9db0bf8f5d9c","Type":"ContainerDied","Data":"6ad30539cd9fc545425e990bc81341e1099e04f3b0dbc4182683f837938c3dbc"} Oct 30 22:00:04.013762 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:04.013448 5748 scope.go:122] "RemoveContainer" containerID="2aa50ea0a4681e07033546044fcc6d217180b2c499f81e0339c0d32c21305550" Oct 30 22:00:04.013762 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:04.013666 5748 scope.go:122] "RemoveContainer" containerID="6ad30539cd9fc545425e990bc81341e1099e04f3b0dbc4182683f837938c3dbc" Oct 30 22:00:04.014075 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:00:04.014043 5748 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-71a3d788-29889960-t7cvk_odh-ai-gateway-infra(3be41157-90f9-4290-b4e3-9db0bf8f5d9c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-71a3d788-29889960-t7cvk" podUID="3be41157-90f9-4290-b4e3-9db0bf8f5d9c" Oct 30 22:00:04.016234 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:04.016211 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91874e-29889960-nj2rv" event={"ID":"6990f1c8-d49d-4151-a79a-e231ca0986a4","Type":"ContainerStarted","Data":"d261b65231c530b47bd9510dca98e337bd30158e37e26fb3fb68a7c1825a8694"} Oct 30 22:00:04.018254 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:04.018227 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2c8f56-29889960-m5257" event={"ID":"da3ff95c-c45b-4f37-a080-9ed58a709e1d","Type":"ContainerStarted","Data":"ca133422cf31404c893280c29d89cfc21eaae117d9a01e01d1bb04e34e595758"} Oct 30 22:00:04.020262 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:04.020242 5748 generic.go:356] "Generic (PLEG): container finished" podID="a9a3ac51-bd06-4343-b765-5d60748fb62d" containerID="6e8e272f8c1e055abd8c2a799fbcabfc30cfb900a4e51373678b1f2a2c2e503b" exitCode=60 Oct 30 22:00:04.020362 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:04.020325 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889960-hm42n" event={"ID":"a9a3ac51-bd06-4343-b765-5d60748fb62d","Type":"ContainerDied","Data":"6e8e272f8c1e055abd8c2a799fbcabfc30cfb900a4e51373678b1f2a2c2e503b"} Oct 30 22:00:04.020523 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:04.020506 5748 scope.go:122] "RemoveContainer" containerID="6e8e272f8c1e055abd8c2a799fbcabfc30cfb900a4e51373678b1f2a2c2e503b" Oct 30 22:00:04.020788 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:00:04.020764 5748 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889960-hm42n_odh-ai-gateway-infra(a9a3ac51-bd06-4343-b765-5d60748fb62d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889960-hm42n" podUID="a9a3ac51-bd06-4343-b765-5d60748fb62d" Oct 30 22:00:04.022285 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:04.022259 5748 generic.go:356] "Generic (PLEG): container finished" podID="67f2eccf-1409-448c-b69d-eb739ab2195e" containerID="4690451fb7c69388388edc0497f29a1947d814a398d25a062b4e3cb330d39d77" exitCode=60 Oct 30 22:00:04.022394 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:04.022333 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c7d477-29889960-l5cjm" event={"ID":"67f2eccf-1409-448c-b69d-eb739ab2195e","Type":"ContainerDied","Data":"4690451fb7c69388388edc0497f29a1947d814a398d25a062b4e3cb330d39d77"} Oct 30 22:00:04.022589 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:04.022574 5748 scope.go:122] "RemoveContainer" containerID="4690451fb7c69388388edc0497f29a1947d814a398d25a062b4e3cb330d39d77" Oct 30 22:00:04.022892 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:00:04.022872 5748 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-c7d477-29889960-l5cjm_odh-ai-gateway-infra(67f2eccf-1409-448c-b69d-eb739ab2195e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c7d477-29889960-l5cjm" podUID="67f2eccf-1409-448c-b69d-eb739ab2195e" Oct 30 22:00:04.024329 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:04.024311 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1de75a-29889960-rl48f" event={"ID":"f46b5473-f092-4bbb-8486-3c2b1da66231","Type":"ContainerStarted","Data":"613b445500934b38f85d54acfdb944d1e9818eacc0d7a32505cc1e883b7e4c5d"} Oct 30 22:00:04.026339 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:04.026316 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ea374-29889960-68g75" event={"ID":"13a18f96-f1ff-436a-8cb9-78d656a45ee4","Type":"ContainerStarted","Data":"d338ec7fc890fe8fb4f64517ea52ddfc2740f57a7472efc544cebcb01720d284"} Oct 30 22:00:04.028596 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:04.028579 5748 scope.go:122] "RemoveContainer" containerID="bf347107617b8330363a16b3f849ef787de26c1c04f8d38bd05b85fdc9481289" Oct 30 22:00:04.041194 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:04.041177 5748 scope.go:122] "RemoveContainer" containerID="c369db16dffe45378d37da632b68b858148316ea42ab3aac5449f39a1fe07510" Oct 30 22:00:04.596653 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:04.596573 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78c632-qnwqv"] Oct 30 22:00:04.605658 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:04.605630 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78c632-qnwqv"] Oct 30 22:00:05.034865 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:05.034836 5748 scope.go:122] "RemoveContainer" containerID="6e8e272f8c1e055abd8c2a799fbcabfc30cfb900a4e51373678b1f2a2c2e503b" Oct 30 22:00:05.035325 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:00:05.035209 5748 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889960-hm42n_odh-ai-gateway-infra(a9a3ac51-bd06-4343-b765-5d60748fb62d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889960-hm42n" podUID="a9a3ac51-bd06-4343-b765-5d60748fb62d" Oct 30 22:00:05.036529 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:05.036505 5748 scope.go:122] "RemoveContainer" containerID="4690451fb7c69388388edc0497f29a1947d814a398d25a062b4e3cb330d39d77" Oct 30 22:00:05.036852 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:00:05.036830 5748 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-c7d477-29889960-l5cjm_odh-ai-gateway-infra(67f2eccf-1409-448c-b69d-eb739ab2195e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c7d477-29889960-l5cjm" podUID="67f2eccf-1409-448c-b69d-eb739ab2195e" Oct 30 22:00:05.038080 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:05.038061 5748 generic.go:356] "Generic (PLEG): container finished" podID="f46b5473-f092-4bbb-8486-3c2b1da66231" containerID="613b445500934b38f85d54acfdb944d1e9818eacc0d7a32505cc1e883b7e4c5d" exitCode=60 Oct 30 22:00:05.038156 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:05.038121 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1de75a-29889960-rl48f" event={"ID":"f46b5473-f092-4bbb-8486-3c2b1da66231","Type":"ContainerDied","Data":"613b445500934b38f85d54acfdb944d1e9818eacc0d7a32505cc1e883b7e4c5d"} Oct 30 22:00:05.038208 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:05.038170 5748 scope.go:122] "RemoveContainer" containerID="5d2652fdf5d7718453b85451cc4e19c21dc13472215f8a648729fab60ef9fbb3" Oct 30 22:00:05.038403 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:05.038377 5748 scope.go:122] "RemoveContainer" containerID="613b445500934b38f85d54acfdb944d1e9818eacc0d7a32505cc1e883b7e4c5d" Oct 30 22:00:05.038708 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:00:05.038685 5748 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-1de75a-29889960-rl48f_odh-ai-gateway-infra(f46b5473-f092-4bbb-8486-3c2b1da66231)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1de75a-29889960-rl48f" podUID="f46b5473-f092-4bbb-8486-3c2b1da66231" Oct 30 22:00:05.039988 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:05.039957 5748 generic.go:356] "Generic (PLEG): container finished" podID="13a18f96-f1ff-436a-8cb9-78d656a45ee4" containerID="d338ec7fc890fe8fb4f64517ea52ddfc2740f57a7472efc544cebcb01720d284" exitCode=60 Oct 30 22:00:05.040085 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:05.040009 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ea374-29889960-68g75" event={"ID":"13a18f96-f1ff-436a-8cb9-78d656a45ee4","Type":"ContainerDied","Data":"d338ec7fc890fe8fb4f64517ea52ddfc2740f57a7472efc544cebcb01720d284"} Oct 30 22:00:05.040238 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:05.040223 5748 scope.go:122] "RemoveContainer" containerID="d338ec7fc890fe8fb4f64517ea52ddfc2740f57a7472efc544cebcb01720d284" Oct 30 22:00:05.040509 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:00:05.040492 5748 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-6ea374-29889960-68g75_odh-ai-gateway-infra(13a18f96-f1ff-436a-8cb9-78d656a45ee4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ea374-29889960-68g75" podUID="13a18f96-f1ff-436a-8cb9-78d656a45ee4" Oct 30 22:00:05.041765 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:05.041748 5748 scope.go:122] "RemoveContainer" containerID="6ad30539cd9fc545425e990bc81341e1099e04f3b0dbc4182683f837938c3dbc" Oct 30 22:00:05.042005 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:00:05.041988 5748 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-71a3d788-29889960-t7cvk_odh-ai-gateway-infra(3be41157-90f9-4290-b4e3-9db0bf8f5d9c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-71a3d788-29889960-t7cvk" podUID="3be41157-90f9-4290-b4e3-9db0bf8f5d9c" Oct 30 22:00:05.043353 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:05.043332 5748 generic.go:356] "Generic (PLEG): container finished" podID="6990f1c8-d49d-4151-a79a-e231ca0986a4" containerID="d261b65231c530b47bd9510dca98e337bd30158e37e26fb3fb68a7c1825a8694" exitCode=60 Oct 30 22:00:05.043437 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:05.043417 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91874e-29889960-nj2rv" event={"ID":"6990f1c8-d49d-4151-a79a-e231ca0986a4","Type":"ContainerDied","Data":"d261b65231c530b47bd9510dca98e337bd30158e37e26fb3fb68a7c1825a8694"} Oct 30 22:00:05.043622 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:05.043604 5748 scope.go:122] "RemoveContainer" containerID="d261b65231c530b47bd9510dca98e337bd30158e37e26fb3fb68a7c1825a8694" Oct 30 22:00:05.043866 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:00:05.043843 5748 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-91874e-29889960-nj2rv_odh-ai-gateway-infra(6990f1c8-d49d-4151-a79a-e231ca0986a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91874e-29889960-nj2rv" podUID="6990f1c8-d49d-4151-a79a-e231ca0986a4" Oct 30 22:00:05.045419 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:05.045399 5748 generic.go:356] "Generic (PLEG): container finished" podID="da3ff95c-c45b-4f37-a080-9ed58a709e1d" containerID="ca133422cf31404c893280c29d89cfc21eaae117d9a01e01d1bb04e34e595758" exitCode=60 Oct 30 22:00:05.045495 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:05.045476 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2c8f56-29889960-m5257" event={"ID":"da3ff95c-c45b-4f37-a080-9ed58a709e1d","Type":"ContainerDied","Data":"ca133422cf31404c893280c29d89cfc21eaae117d9a01e01d1bb04e34e595758"} Oct 30 22:00:05.045619 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:05.045606 5748 scope.go:122] "RemoveContainer" containerID="ca133422cf31404c893280c29d89cfc21eaae117d9a01e01d1bb04e34e595758" Oct 30 22:00:05.045830 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:00:05.045815 5748 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-2c8f56-29889960-m5257_odh-ai-gateway-infra(da3ff95c-c45b-4f37-a080-9ed58a709e1d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2c8f56-29889960-m5257" podUID="da3ff95c-c45b-4f37-a080-9ed58a709e1d" Oct 30 22:00:05.051313 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:05.051296 5748 scope.go:122] "RemoveContainer" containerID="1bfb723049732074b9e2ee80fd2b96c5998ca99f749e6fe8a1819f463f766c22" Oct 30 22:00:05.064052 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:05.064038 5748 scope.go:122] "RemoveContainer" containerID="784ed1aac7174d304e4198986737aa48393b67ee7ba3b879ddd94863577db777" Oct 30 22:00:05.076328 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:05.076311 5748 scope.go:122] "RemoveContainer" containerID="d01f8c8127952e95ccfe1d55ddc33eb10bbedfdf297f87b6a31ccd8060c82a58" Oct 30 22:00:06.053726 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:06.053698 5748 scope.go:122] "RemoveContainer" containerID="d261b65231c530b47bd9510dca98e337bd30158e37e26fb3fb68a7c1825a8694" Oct 30 22:00:06.054218 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:00:06.054077 5748 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-91874e-29889960-nj2rv_odh-ai-gateway-infra(6990f1c8-d49d-4151-a79a-e231ca0986a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91874e-29889960-nj2rv" podUID="6990f1c8-d49d-4151-a79a-e231ca0986a4" Oct 30 22:00:06.055339 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:06.055318 5748 scope.go:122] "RemoveContainer" containerID="ca133422cf31404c893280c29d89cfc21eaae117d9a01e01d1bb04e34e595758" Oct 30 22:00:06.055613 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:00:06.055592 5748 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-2c8f56-29889960-m5257_odh-ai-gateway-infra(da3ff95c-c45b-4f37-a080-9ed58a709e1d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2c8f56-29889960-m5257" podUID="da3ff95c-c45b-4f37-a080-9ed58a709e1d" Oct 30 22:00:06.056955 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:06.056931 5748 scope.go:122] "RemoveContainer" containerID="613b445500934b38f85d54acfdb944d1e9818eacc0d7a32505cc1e883b7e4c5d" Oct 30 22:00:06.057178 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:00:06.057160 5748 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-1de75a-29889960-rl48f_odh-ai-gateway-infra(f46b5473-f092-4bbb-8486-3c2b1da66231)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1de75a-29889960-rl48f" podUID="f46b5473-f092-4bbb-8486-3c2b1da66231" Oct 30 22:00:06.058414 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:06.058395 5748 scope.go:122] "RemoveContainer" containerID="d338ec7fc890fe8fb4f64517ea52ddfc2740f57a7472efc544cebcb01720d284" Oct 30 22:00:06.058600 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:00:06.058583 5748 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-6ea374-29889960-68g75_odh-ai-gateway-infra(13a18f96-f1ff-436a-8cb9-78d656a45ee4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ea374-29889960-68g75" podUID="13a18f96-f1ff-436a-8cb9-78d656a45ee4" Oct 30 22:00:06.423223 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:06.423189 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="370cd288-ede4-49c6-96f2-d7417b877384" path="/var/lib/kubelet/pods/370cd288-ede4-49c6-96f2-d7417b877384/volumes" Oct 30 22:00:06.642137 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:06.642096 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl"] Oct 30 22:00:06.642432 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:06.642397 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl" podUID="e231fcd9-6b46-4f08-bed5-947a82dc6d03" containerName="istio-proxy" containerID="cri-o://0823e1f37d92dc72d7438c8313bf2fbb33675196767e4e28a0de8467e861c3a6" gracePeriod=30 Oct 30 22:00:11.981053 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:11.981024 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl" Oct 30 22:00:12.054031 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:12.053997 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e231fcd9-6b46-4f08-bed5-947a82dc6d03-istio-token\" (UniqueName: \"kubernetes.io/projected/e231fcd9-6b46-4f08-bed5-947a82dc6d03-istio-token\") pod \"e231fcd9-6b46-4f08-bed5-947a82dc6d03\" (UID: \"e231fcd9-6b46-4f08-bed5-947a82dc6d03\") " Oct 30 22:00:12.054219 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:12.054064 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e231fcd9-6b46-4f08-bed5-947a82dc6d03-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e231fcd9-6b46-4f08-bed5-947a82dc6d03-istio-podinfo\") pod \"e231fcd9-6b46-4f08-bed5-947a82dc6d03\" (UID: \"e231fcd9-6b46-4f08-bed5-947a82dc6d03\") " Oct 30 22:00:12.054219 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:12.054108 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e231fcd9-6b46-4f08-bed5-947a82dc6d03-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e231fcd9-6b46-4f08-bed5-947a82dc6d03-istio-envoy\") pod \"e231fcd9-6b46-4f08-bed5-947a82dc6d03\" (UID: \"e231fcd9-6b46-4f08-bed5-947a82dc6d03\") " Oct 30 22:00:12.054219 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:12.054136 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e231fcd9-6b46-4f08-bed5-947a82dc6d03-kube-api-access-tj79s\" (UniqueName: \"kubernetes.io/projected/e231fcd9-6b46-4f08-bed5-947a82dc6d03-kube-api-access-tj79s\") pod \"e231fcd9-6b46-4f08-bed5-947a82dc6d03\" (UID: \"e231fcd9-6b46-4f08-bed5-947a82dc6d03\") " Oct 30 22:00:12.054395 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:12.054240 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e231fcd9-6b46-4f08-bed5-947a82dc6d03-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e231fcd9-6b46-4f08-bed5-947a82dc6d03-istio-data\") pod \"e231fcd9-6b46-4f08-bed5-947a82dc6d03\" (UID: \"e231fcd9-6b46-4f08-bed5-947a82dc6d03\") " Oct 30 22:00:12.054395 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:12.054311 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e231fcd9-6b46-4f08-bed5-947a82dc6d03-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e231fcd9-6b46-4f08-bed5-947a82dc6d03-istiod-ca-cert\") pod \"e231fcd9-6b46-4f08-bed5-947a82dc6d03\" (UID: \"e231fcd9-6b46-4f08-bed5-947a82dc6d03\") " Oct 30 22:00:12.054395 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:12.054352 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e231fcd9-6b46-4f08-bed5-947a82dc6d03-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e231fcd9-6b46-4f08-bed5-947a82dc6d03-workload-certs\") pod \"e231fcd9-6b46-4f08-bed5-947a82dc6d03\" (UID: \"e231fcd9-6b46-4f08-bed5-947a82dc6d03\") " Oct 30 22:00:12.054570 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:12.054542 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e231fcd9-6b46-4f08-bed5-947a82dc6d03-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e231fcd9-6b46-4f08-bed5-947a82dc6d03-workload-socket\") pod \"e231fcd9-6b46-4f08-bed5-947a82dc6d03\" (UID: \"e231fcd9-6b46-4f08-bed5-947a82dc6d03\") " Oct 30 22:00:12.054629 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:12.054587 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e231fcd9-6b46-4f08-bed5-947a82dc6d03-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e231fcd9-6b46-4f08-bed5-947a82dc6d03-credential-socket\") pod \"e231fcd9-6b46-4f08-bed5-947a82dc6d03\" (UID: \"e231fcd9-6b46-4f08-bed5-947a82dc6d03\") " Oct 30 22:00:12.054731 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:12.054696 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e231fcd9-6b46-4f08-bed5-947a82dc6d03-istio-data" pod "e231fcd9-6b46-4f08-bed5-947a82dc6d03" (UID: "e231fcd9-6b46-4f08-bed5-947a82dc6d03"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:00:12.054978 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:12.054953 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e231fcd9-6b46-4f08-bed5-947a82dc6d03-workload-socket" pod "e231fcd9-6b46-4f08-bed5-947a82dc6d03" (UID: "e231fcd9-6b46-4f08-bed5-947a82dc6d03"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:00:12.055078 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:12.054981 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e231fcd9-6b46-4f08-bed5-947a82dc6d03-workload-certs" pod "e231fcd9-6b46-4f08-bed5-947a82dc6d03" (UID: "e231fcd9-6b46-4f08-bed5-947a82dc6d03"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:00:12.055217 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:12.055179 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e231fcd9-6b46-4f08-bed5-947a82dc6d03-credential-socket" pod "e231fcd9-6b46-4f08-bed5-947a82dc6d03" (UID: "e231fcd9-6b46-4f08-bed5-947a82dc6d03"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:00:12.055315 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:12.055292 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e231fcd9-6b46-4f08-bed5-947a82dc6d03-istiod-ca-cert" pod "e231fcd9-6b46-4f08-bed5-947a82dc6d03" (UID: "e231fcd9-6b46-4f08-bed5-947a82dc6d03"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:12.055516 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:12.055494 5748 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e231fcd9-6b46-4f08-bed5-947a82dc6d03-workload-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:12.055597 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:12.055525 5748 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e231fcd9-6b46-4f08-bed5-947a82dc6d03-credential-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:12.055597 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:12.055540 5748 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e231fcd9-6b46-4f08-bed5-947a82dc6d03-istio-data\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:12.055597 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:12.055554 5748 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e231fcd9-6b46-4f08-bed5-947a82dc6d03-istiod-ca-cert\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:12.055597 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:12.055569 5748 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e231fcd9-6b46-4f08-bed5-947a82dc6d03-workload-certs\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:12.056584 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:12.056546 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e231fcd9-6b46-4f08-bed5-947a82dc6d03-istio-podinfo" pod "e231fcd9-6b46-4f08-bed5-947a82dc6d03" (UID: "e231fcd9-6b46-4f08-bed5-947a82dc6d03"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:00:12.056797 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:12.056772 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e231fcd9-6b46-4f08-bed5-947a82dc6d03-istio-envoy" pod "e231fcd9-6b46-4f08-bed5-947a82dc6d03" (UID: "e231fcd9-6b46-4f08-bed5-947a82dc6d03"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:00:12.057051 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:12.057030 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e231fcd9-6b46-4f08-bed5-947a82dc6d03-istio-token" pod "e231fcd9-6b46-4f08-bed5-947a82dc6d03" (UID: "e231fcd9-6b46-4f08-bed5-947a82dc6d03"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:12.059036 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:12.059017 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e231fcd9-6b46-4f08-bed5-947a82dc6d03-kube-api-access-tj79s" pod "e231fcd9-6b46-4f08-bed5-947a82dc6d03" (UID: "e231fcd9-6b46-4f08-bed5-947a82dc6d03"). InnerVolumeSpecName "kube-api-access-tj79s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:12.133125 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:12.133094 5748 generic.go:356] "Generic (PLEG): container finished" podID="e231fcd9-6b46-4f08-bed5-947a82dc6d03" containerID="0823e1f37d92dc72d7438c8313bf2fbb33675196767e4e28a0de8467e861c3a6" exitCode=0 Oct 30 22:00:12.133290 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:12.133151 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl" Oct 30 22:00:12.133290 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:12.133180 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl" event={"ID":"e231fcd9-6b46-4f08-bed5-947a82dc6d03","Type":"ContainerDied","Data":"0823e1f37d92dc72d7438c8313bf2fbb33675196767e4e28a0de8467e861c3a6"} Oct 30 22:00:12.133290 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:12.133237 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl" event={"ID":"e231fcd9-6b46-4f08-bed5-947a82dc6d03","Type":"ContainerDied","Data":"bcf63c9b6caf1f34f68f4756735b77de3231ee408d302caa43b6115f22577559"} Oct 30 22:00:12.133290 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:12.133266 5748 scope.go:122] "RemoveContainer" containerID="0823e1f37d92dc72d7438c8313bf2fbb33675196767e4e28a0de8467e861c3a6" Oct 30 22:00:12.146180 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:12.146161 5748 scope.go:122] "RemoveContainer" containerID="0823e1f37d92dc72d7438c8313bf2fbb33675196767e4e28a0de8467e861c3a6" Oct 30 22:00:12.146455 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:00:12.146434 5748 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0823e1f37d92dc72d7438c8313bf2fbb33675196767e4e28a0de8467e861c3a6\": container with ID starting with 0823e1f37d92dc72d7438c8313bf2fbb33675196767e4e28a0de8467e861c3a6 not found: ID does not exist" containerID="0823e1f37d92dc72d7438c8313bf2fbb33675196767e4e28a0de8467e861c3a6" Oct 30 22:00:12.146504 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:12.146464 5748 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0823e1f37d92dc72d7438c8313bf2fbb33675196767e4e28a0de8467e861c3a6"} err="failed to get container status \"0823e1f37d92dc72d7438c8313bf2fbb33675196767e4e28a0de8467e861c3a6\": rpc error: code = NotFound desc = could not find container \"0823e1f37d92dc72d7438c8313bf2fbb33675196767e4e28a0de8467e861c3a6\": container with ID starting with 0823e1f37d92dc72d7438c8313bf2fbb33675196767e4e28a0de8467e861c3a6 not found: ID does not exist" Oct 30 22:00:12.157312 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:12.157286 5748 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e231fcd9-6b46-4f08-bed5-947a82dc6d03-istio-token\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:12.157312 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:12.157310 5748 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e231fcd9-6b46-4f08-bed5-947a82dc6d03-istio-podinfo\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:12.157474 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:12.157320 5748 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e231fcd9-6b46-4f08-bed5-947a82dc6d03-istio-envoy\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:12.157474 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:12.157330 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tj79s\" (UniqueName: \"kubernetes.io/projected/e231fcd9-6b46-4f08-bed5-947a82dc6d03-kube-api-access-tj79s\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:12.282973 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:12.282862 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl"] Oct 30 22:00:12.287263 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:12.287233 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-48ssl"] Oct 30 22:00:12.418193 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:12.418152 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e231fcd9-6b46-4f08-bed5-947a82dc6d03" path="/var/lib/kubelet/pods/e231fcd9-6b46-4f08-bed5-947a82dc6d03/volumes" Oct 30 22:00:13.061094 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:13.061062 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5"] Oct 30 22:00:13.061556 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:13.061296 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5" podUID="1faeab78-65f0-489f-9aeb-d651dd7cac1b" containerName="istio-proxy" containerID="cri-o://7161b894c7d64d0734bbfca6ed030cf9f8bcd31da3cbedddde8276f67088db2c" gracePeriod=30 Oct 30 22:00:15.410151 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:15.410118 5748 scope.go:122] "RemoveContainer" containerID="d261b65231c530b47bd9510dca98e337bd30158e37e26fb3fb68a7c1825a8694" Oct 30 22:00:15.410648 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:15.410214 5748 scope.go:122] "RemoveContainer" containerID="ca133422cf31404c893280c29d89cfc21eaae117d9a01e01d1bb04e34e595758" Oct 30 22:00:16.175960 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:16.175921 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91874e-29889960-nj2rv" event={"ID":"6990f1c8-d49d-4151-a79a-e231ca0986a4","Type":"ContainerStarted","Data":"982d146746789f68c6b1c874db2ef2a0414ecbb344c82dd5602763da3c5b6f13"} Oct 30 22:00:16.178358 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:16.178328 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2c8f56-29889960-m5257" event={"ID":"da3ff95c-c45b-4f37-a080-9ed58a709e1d","Type":"ContainerStarted","Data":"5ce3006e5b7ce63d605894940d290ce794fbbe68dfdf089a7264e19b53f568a2"} Oct 30 22:00:16.410348 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:16.410319 5748 scope.go:122] "RemoveContainer" containerID="d338ec7fc890fe8fb4f64517ea52ddfc2740f57a7472efc544cebcb01720d284" Oct 30 22:00:16.411663 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:16.411282 5748 scope.go:122] "RemoveContainer" containerID="4690451fb7c69388388edc0497f29a1947d814a398d25a062b4e3cb330d39d77" Oct 30 22:00:17.186941 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:17.186905 5748 generic.go:356] "Generic (PLEG): container finished" podID="6990f1c8-d49d-4151-a79a-e231ca0986a4" containerID="982d146746789f68c6b1c874db2ef2a0414ecbb344c82dd5602763da3c5b6f13" exitCode=60 Oct 30 22:00:17.187111 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:17.186945 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91874e-29889960-nj2rv" event={"ID":"6990f1c8-d49d-4151-a79a-e231ca0986a4","Type":"ContainerDied","Data":"982d146746789f68c6b1c874db2ef2a0414ecbb344c82dd5602763da3c5b6f13"} Oct 30 22:00:17.187111 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:17.187003 5748 scope.go:122] "RemoveContainer" containerID="d261b65231c530b47bd9510dca98e337bd30158e37e26fb3fb68a7c1825a8694" Oct 30 22:00:17.187422 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:17.187406 5748 scope.go:122] "RemoveContainer" containerID="982d146746789f68c6b1c874db2ef2a0414ecbb344c82dd5602763da3c5b6f13" Oct 30 22:00:17.187761 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:00:17.187737 5748 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-91874e-29889960-nj2rv_odh-ai-gateway-infra(6990f1c8-d49d-4151-a79a-e231ca0986a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91874e-29889960-nj2rv" podUID="6990f1c8-d49d-4151-a79a-e231ca0986a4" Oct 30 22:00:17.189077 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:17.189056 5748 generic.go:356] "Generic (PLEG): container finished" podID="da3ff95c-c45b-4f37-a080-9ed58a709e1d" containerID="5ce3006e5b7ce63d605894940d290ce794fbbe68dfdf089a7264e19b53f568a2" exitCode=60 Oct 30 22:00:17.189156 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:17.189129 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2c8f56-29889960-m5257" event={"ID":"da3ff95c-c45b-4f37-a080-9ed58a709e1d","Type":"ContainerDied","Data":"5ce3006e5b7ce63d605894940d290ce794fbbe68dfdf089a7264e19b53f568a2"} Oct 30 22:00:17.189492 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:17.189477 5748 scope.go:122] "RemoveContainer" containerID="5ce3006e5b7ce63d605894940d290ce794fbbe68dfdf089a7264e19b53f568a2" Oct 30 22:00:17.189808 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:00:17.189790 5748 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-2c8f56-29889960-m5257_odh-ai-gateway-infra(da3ff95c-c45b-4f37-a080-9ed58a709e1d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2c8f56-29889960-m5257" podUID="da3ff95c-c45b-4f37-a080-9ed58a709e1d" Oct 30 22:00:17.191151 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:17.191127 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c7d477-29889960-l5cjm" event={"ID":"67f2eccf-1409-448c-b69d-eb739ab2195e","Type":"ContainerStarted","Data":"0e826933a9d3f1533250091500de93f9ab97339e45e00a3517f79a869b3d47a3"} Oct 30 22:00:17.193049 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:17.193028 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ea374-29889960-68g75" event={"ID":"13a18f96-f1ff-436a-8cb9-78d656a45ee4","Type":"ContainerStarted","Data":"ff9e95ba14eb04859bb299ef5555406f9aa01c676ee9afd443f14d8ae7c3642a"} Oct 30 22:00:17.199994 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:17.199971 5748 scope.go:122] "RemoveContainer" containerID="ca133422cf31404c893280c29d89cfc21eaae117d9a01e01d1bb04e34e595758" Oct 30 22:00:17.307627 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:17.307592 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91874e-29889960-nj2rv"] Oct 30 22:00:17.410644 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:17.410607 5748 scope.go:122] "RemoveContainer" containerID="613b445500934b38f85d54acfdb944d1e9818eacc0d7a32505cc1e883b7e4c5d" Oct 30 22:00:17.411192 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:17.410769 5748 scope.go:122] "RemoveContainer" containerID="6e8e272f8c1e055abd8c2a799fbcabfc30cfb900a4e51373678b1f2a2c2e503b" Oct 30 22:00:17.411332 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:17.411308 5748 scope.go:122] "RemoveContainer" containerID="6ad30539cd9fc545425e990bc81341e1099e04f3b0dbc4182683f837938c3dbc" Oct 30 22:00:17.424129 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:17.424079 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2c8f56-29889960-m5257"] Oct 30 22:00:18.204251 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:18.204211 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889960-hm42n" event={"ID":"a9a3ac51-bd06-4343-b765-5d60748fb62d","Type":"ContainerStarted","Data":"e16940b67f2bee99e9569404851d98cd362c74689917b96f004bb1b6203d9bbf"} Oct 30 22:00:18.207465 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:18.207436 5748 generic.go:356] "Generic (PLEG): container finished" podID="67f2eccf-1409-448c-b69d-eb739ab2195e" containerID="0e826933a9d3f1533250091500de93f9ab97339e45e00a3517f79a869b3d47a3" exitCode=60 Oct 30 22:00:18.207603 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:18.207504 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c7d477-29889960-l5cjm" event={"ID":"67f2eccf-1409-448c-b69d-eb739ab2195e","Type":"ContainerDied","Data":"0e826933a9d3f1533250091500de93f9ab97339e45e00a3517f79a869b3d47a3"} Oct 30 22:00:18.207603 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:18.207559 5748 scope.go:122] "RemoveContainer" containerID="4690451fb7c69388388edc0497f29a1947d814a398d25a062b4e3cb330d39d77" Oct 30 22:00:18.207959 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:18.207943 5748 scope.go:122] "RemoveContainer" containerID="0e826933a9d3f1533250091500de93f9ab97339e45e00a3517f79a869b3d47a3" Oct 30 22:00:18.208282 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:00:18.208257 5748 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-c7d477-29889960-l5cjm_odh-ai-gateway-infra(67f2eccf-1409-448c-b69d-eb739ab2195e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c7d477-29889960-l5cjm" podUID="67f2eccf-1409-448c-b69d-eb739ab2195e" Oct 30 22:00:18.209659 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:18.209635 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1de75a-29889960-rl48f" event={"ID":"f46b5473-f092-4bbb-8486-3c2b1da66231","Type":"ContainerStarted","Data":"ec34df06e9fceb3cf267e2be50adb14a3b1f5e6439c4c7a53795dc2b56151543"} Oct 30 22:00:18.222565 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:18.222536 5748 generic.go:356] "Generic (PLEG): container finished" podID="13a18f96-f1ff-436a-8cb9-78d656a45ee4" containerID="ff9e95ba14eb04859bb299ef5555406f9aa01c676ee9afd443f14d8ae7c3642a" exitCode=60 Oct 30 22:00:18.222710 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:18.222611 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ea374-29889960-68g75" event={"ID":"13a18f96-f1ff-436a-8cb9-78d656a45ee4","Type":"ContainerDied","Data":"ff9e95ba14eb04859bb299ef5555406f9aa01c676ee9afd443f14d8ae7c3642a"} Oct 30 22:00:18.223074 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:18.223055 5748 scope.go:122] "RemoveContainer" containerID="ff9e95ba14eb04859bb299ef5555406f9aa01c676ee9afd443f14d8ae7c3642a" Oct 30 22:00:18.223318 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:00:18.223296 5748 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-6ea374-29889960-68g75_odh-ai-gateway-infra(13a18f96-f1ff-436a-8cb9-78d656a45ee4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ea374-29889960-68g75" podUID="13a18f96-f1ff-436a-8cb9-78d656a45ee4" Oct 30 22:00:18.224956 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:18.224933 5748 generic.go:356] "Generic (PLEG): container finished" podID="1faeab78-65f0-489f-9aeb-d651dd7cac1b" containerID="7161b894c7d64d0734bbfca6ed030cf9f8bcd31da3cbedddde8276f67088db2c" exitCode=0 Oct 30 22:00:18.225042 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:18.224968 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5" event={"ID":"1faeab78-65f0-489f-9aeb-d651dd7cac1b","Type":"ContainerDied","Data":"7161b894c7d64d0734bbfca6ed030cf9f8bcd31da3cbedddde8276f67088db2c"} Oct 30 22:00:18.227071 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:18.227049 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-71a3d788-29889960-t7cvk" event={"ID":"3be41157-90f9-4290-b4e3-9db0bf8f5d9c","Type":"ContainerStarted","Data":"c28a178840b85be19a4ac6601b1fbb75708d0e84d9bdc0fc7d3c95cc35fa4d5c"} Oct 30 22:00:18.232659 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:18.232635 5748 scope.go:122] "RemoveContainer" containerID="d338ec7fc890fe8fb4f64517ea52ddfc2740f57a7472efc544cebcb01720d284" Oct 30 22:00:18.273515 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:18.272978 5748 scope.go:122] "RemoveContainer" containerID="afa4d973d6d47fbdd9c9b149a638994e3606321fc430e0e577c5e9ab84c942c9" Oct 30 22:00:18.418108 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:18.418061 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5" Oct 30 22:00:18.460824 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:18.460791 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c7d477-29889960-l5cjm"] Oct 30 22:00:18.574794 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:18.574722 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1faeab78-65f0-489f-9aeb-d651dd7cac1b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1faeab78-65f0-489f-9aeb-d651dd7cac1b-credential-socket\") pod \"1faeab78-65f0-489f-9aeb-d651dd7cac1b\" (UID: \"1faeab78-65f0-489f-9aeb-d651dd7cac1b\") " Oct 30 22:00:18.574794 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:18.574770 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1faeab78-65f0-489f-9aeb-d651dd7cac1b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1faeab78-65f0-489f-9aeb-d651dd7cac1b-workload-socket\") pod \"1faeab78-65f0-489f-9aeb-d651dd7cac1b\" (UID: \"1faeab78-65f0-489f-9aeb-d651dd7cac1b\") " Oct 30 22:00:18.575018 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:18.574809 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1faeab78-65f0-489f-9aeb-d651dd7cac1b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1faeab78-65f0-489f-9aeb-d651dd7cac1b-istiod-ca-cert\") pod \"1faeab78-65f0-489f-9aeb-d651dd7cac1b\" (UID: \"1faeab78-65f0-489f-9aeb-d651dd7cac1b\") " Oct 30 22:00:18.575018 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:18.574839 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1faeab78-65f0-489f-9aeb-d651dd7cac1b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1faeab78-65f0-489f-9aeb-d651dd7cac1b-istio-envoy\") pod \"1faeab78-65f0-489f-9aeb-d651dd7cac1b\" (UID: \"1faeab78-65f0-489f-9aeb-d651dd7cac1b\") " Oct 30 22:00:18.575018 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:18.574961 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1faeab78-65f0-489f-9aeb-d651dd7cac1b-kube-api-access-ph2m2\" (UniqueName: \"kubernetes.io/projected/1faeab78-65f0-489f-9aeb-d651dd7cac1b-kube-api-access-ph2m2\") pod \"1faeab78-65f0-489f-9aeb-d651dd7cac1b\" (UID: \"1faeab78-65f0-489f-9aeb-d651dd7cac1b\") " Oct 30 22:00:18.575176 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:18.575007 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1faeab78-65f0-489f-9aeb-d651dd7cac1b-credential-socket" pod "1faeab78-65f0-489f-9aeb-d651dd7cac1b" (UID: "1faeab78-65f0-489f-9aeb-d651dd7cac1b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:00:18.575176 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:18.575067 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1faeab78-65f0-489f-9aeb-d651dd7cac1b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1faeab78-65f0-489f-9aeb-d651dd7cac1b-workload-certs\") pod \"1faeab78-65f0-489f-9aeb-d651dd7cac1b\" (UID: \"1faeab78-65f0-489f-9aeb-d651dd7cac1b\") " Oct 30 22:00:18.575176 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:18.575110 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1faeab78-65f0-489f-9aeb-d651dd7cac1b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1faeab78-65f0-489f-9aeb-d651dd7cac1b-istio-data\") pod \"1faeab78-65f0-489f-9aeb-d651dd7cac1b\" (UID: \"1faeab78-65f0-489f-9aeb-d651dd7cac1b\") " Oct 30 22:00:18.575176 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:18.575157 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1faeab78-65f0-489f-9aeb-d651dd7cac1b-workload-socket" pod "1faeab78-65f0-489f-9aeb-d651dd7cac1b" (UID: "1faeab78-65f0-489f-9aeb-d651dd7cac1b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:00:18.575374 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:18.575218 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1faeab78-65f0-489f-9aeb-d651dd7cac1b-istio-token\" (UniqueName: \"kubernetes.io/projected/1faeab78-65f0-489f-9aeb-d651dd7cac1b-istio-token\") pod \"1faeab78-65f0-489f-9aeb-d651dd7cac1b\" (UID: \"1faeab78-65f0-489f-9aeb-d651dd7cac1b\") " Oct 30 22:00:18.575374 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:18.575238 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1faeab78-65f0-489f-9aeb-d651dd7cac1b-istiod-ca-cert" pod "1faeab78-65f0-489f-9aeb-d651dd7cac1b" (UID: "1faeab78-65f0-489f-9aeb-d651dd7cac1b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:18.575374 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:18.575282 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1faeab78-65f0-489f-9aeb-d651dd7cac1b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1faeab78-65f0-489f-9aeb-d651dd7cac1b-istio-podinfo\") pod \"1faeab78-65f0-489f-9aeb-d651dd7cac1b\" (UID: \"1faeab78-65f0-489f-9aeb-d651dd7cac1b\") " Oct 30 22:00:18.575528 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:18.575394 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1faeab78-65f0-489f-9aeb-d651dd7cac1b-istio-data" pod "1faeab78-65f0-489f-9aeb-d651dd7cac1b" (UID: "1faeab78-65f0-489f-9aeb-d651dd7cac1b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:00:18.575528 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:18.575483 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1faeab78-65f0-489f-9aeb-d651dd7cac1b-workload-certs" pod "1faeab78-65f0-489f-9aeb-d651dd7cac1b" (UID: "1faeab78-65f0-489f-9aeb-d651dd7cac1b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:00:18.575864 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:18.575839 5748 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1faeab78-65f0-489f-9aeb-d651dd7cac1b-workload-certs\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:18.576035 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:18.575868 5748 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1faeab78-65f0-489f-9aeb-d651dd7cac1b-istio-data\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:18.576035 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:18.575883 5748 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1faeab78-65f0-489f-9aeb-d651dd7cac1b-credential-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:18.576035 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:18.575912 5748 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1faeab78-65f0-489f-9aeb-d651dd7cac1b-workload-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:18.576035 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:18.575943 5748 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1faeab78-65f0-489f-9aeb-d651dd7cac1b-istiod-ca-cert\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:18.577705 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:18.577679 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1faeab78-65f0-489f-9aeb-d651dd7cac1b-kube-api-access-ph2m2" pod "1faeab78-65f0-489f-9aeb-d651dd7cac1b" (UID: "1faeab78-65f0-489f-9aeb-d651dd7cac1b"). InnerVolumeSpecName "kube-api-access-ph2m2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:18.577959 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:18.577933 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1faeab78-65f0-489f-9aeb-d651dd7cac1b-istio-token" pod "1faeab78-65f0-489f-9aeb-d651dd7cac1b" (UID: "1faeab78-65f0-489f-9aeb-d651dd7cac1b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:18.578040 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:18.577964 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1faeab78-65f0-489f-9aeb-d651dd7cac1b-istio-podinfo" pod "1faeab78-65f0-489f-9aeb-d651dd7cac1b" (UID: "1faeab78-65f0-489f-9aeb-d651dd7cac1b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:00:18.578040 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:18.577977 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1faeab78-65f0-489f-9aeb-d651dd7cac1b-istio-envoy" pod "1faeab78-65f0-489f-9aeb-d651dd7cac1b" (UID: "1faeab78-65f0-489f-9aeb-d651dd7cac1b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:00:18.588005 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:18.587980 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ea374-29889960-68g75"] Oct 30 22:00:18.677484 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:18.677449 5748 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1faeab78-65f0-489f-9aeb-d651dd7cac1b-istio-token\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:18.677484 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:18.677486 5748 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1faeab78-65f0-489f-9aeb-d651dd7cac1b-istio-podinfo\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:18.677700 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:18.677502 5748 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1faeab78-65f0-489f-9aeb-d651dd7cac1b-istio-envoy\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:18.677700 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:18.677516 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph2m2\" (UniqueName: \"kubernetes.io/projected/1faeab78-65f0-489f-9aeb-d651dd7cac1b-kube-api-access-ph2m2\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:18.696990 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:18.696961 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2c8f56-29889960-m5257" Oct 30 22:00:18.700490 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:18.700466 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91874e-29889960-nj2rv" Oct 30 22:00:18.880176 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:18.880097 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6990f1c8-d49d-4151-a79a-e231ca0986a4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6990f1c8-d49d-4151-a79a-e231ca0986a4-maas-api-service-ca\") pod \"6990f1c8-d49d-4151-a79a-e231ca0986a4\" (UID: \"6990f1c8-d49d-4151-a79a-e231ca0986a4\") " Oct 30 22:00:18.880176 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:18.880160 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da3ff95c-c45b-4f37-a080-9ed58a709e1d-kube-api-access-9wqwp\" (UniqueName: \"kubernetes.io/projected/da3ff95c-c45b-4f37-a080-9ed58a709e1d-kube-api-access-9wqwp\") pod \"da3ff95c-c45b-4f37-a080-9ed58a709e1d\" (UID: \"da3ff95c-c45b-4f37-a080-9ed58a709e1d\") " Oct 30 22:00:18.880421 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:18.880186 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6990f1c8-d49d-4151-a79a-e231ca0986a4-kube-api-access-tbm2k\" (UniqueName: \"kubernetes.io/projected/6990f1c8-d49d-4151-a79a-e231ca0986a4-kube-api-access-tbm2k\") pod \"6990f1c8-d49d-4151-a79a-e231ca0986a4\" (UID: \"6990f1c8-d49d-4151-a79a-e231ca0986a4\") " Oct 30 22:00:18.880421 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:18.880238 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da3ff95c-c45b-4f37-a080-9ed58a709e1d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da3ff95c-c45b-4f37-a080-9ed58a709e1d-maas-api-service-ca\") pod \"da3ff95c-c45b-4f37-a080-9ed58a709e1d\" (UID: \"da3ff95c-c45b-4f37-a080-9ed58a709e1d\") " Oct 30 22:00:18.880571 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:18.880543 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6990f1c8-d49d-4151-a79a-e231ca0986a4-maas-api-service-ca" pod "6990f1c8-d49d-4151-a79a-e231ca0986a4" (UID: "6990f1c8-d49d-4151-a79a-e231ca0986a4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:18.880643 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:18.880589 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da3ff95c-c45b-4f37-a080-9ed58a709e1d-maas-api-service-ca" pod "da3ff95c-c45b-4f37-a080-9ed58a709e1d" (UID: "da3ff95c-c45b-4f37-a080-9ed58a709e1d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:18.880927 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:18.880884 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da3ff95c-c45b-4f37-a080-9ed58a709e1d-maas-api-service-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:18.880927 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:18.880924 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6990f1c8-d49d-4151-a79a-e231ca0986a4-maas-api-service-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:18.882412 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:18.882393 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da3ff95c-c45b-4f37-a080-9ed58a709e1d-kube-api-access-9wqwp" pod "da3ff95c-c45b-4f37-a080-9ed58a709e1d" (UID: "da3ff95c-c45b-4f37-a080-9ed58a709e1d"). InnerVolumeSpecName "kube-api-access-9wqwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:18.882510 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:18.882488 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6990f1c8-d49d-4151-a79a-e231ca0986a4-kube-api-access-tbm2k" pod "6990f1c8-d49d-4151-a79a-e231ca0986a4" (UID: "6990f1c8-d49d-4151-a79a-e231ca0986a4"). InnerVolumeSpecName "kube-api-access-tbm2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:18.982167 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:18.982136 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wqwp\" (UniqueName: \"kubernetes.io/projected/da3ff95c-c45b-4f37-a080-9ed58a709e1d-kube-api-access-9wqwp\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:18.982167 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:18.982164 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbm2k\" (UniqueName: \"kubernetes.io/projected/6990f1c8-d49d-4151-a79a-e231ca0986a4-kube-api-access-tbm2k\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:19.242093 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:19.242056 5748 generic.go:356] "Generic (PLEG): container finished" podID="f46b5473-f092-4bbb-8486-3c2b1da66231" containerID="ec34df06e9fceb3cf267e2be50adb14a3b1f5e6439c4c7a53795dc2b56151543" exitCode=60 Oct 30 22:00:19.242245 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:19.242144 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1de75a-29889960-rl48f" event={"ID":"f46b5473-f092-4bbb-8486-3c2b1da66231","Type":"ContainerDied","Data":"ec34df06e9fceb3cf267e2be50adb14a3b1f5e6439c4c7a53795dc2b56151543"} Oct 30 22:00:19.242245 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:19.242198 5748 scope.go:122] "RemoveContainer" containerID="613b445500934b38f85d54acfdb944d1e9818eacc0d7a32505cc1e883b7e4c5d" Oct 30 22:00:19.242699 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:19.242674 5748 scope.go:122] "RemoveContainer" containerID="ec34df06e9fceb3cf267e2be50adb14a3b1f5e6439c4c7a53795dc2b56151543" Oct 30 22:00:19.243085 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:00:19.243047 5748 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-1de75a-29889960-rl48f_odh-ai-gateway-infra(f46b5473-f092-4bbb-8486-3c2b1da66231)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1de75a-29889960-rl48f" podUID="f46b5473-f092-4bbb-8486-3c2b1da66231" Oct 30 22:00:19.248511 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:19.248479 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5" event={"ID":"1faeab78-65f0-489f-9aeb-d651dd7cac1b","Type":"ContainerDied","Data":"84ad385b0c19d90ea248f93fc2c6e7656157dc032d801ce21bf2ab493e024148"} Oct 30 22:00:19.249011 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:19.248834 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5" Oct 30 22:00:19.253581 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:19.252818 5748 generic.go:356] "Generic (PLEG): container finished" podID="3be41157-90f9-4290-b4e3-9db0bf8f5d9c" containerID="c28a178840b85be19a4ac6601b1fbb75708d0e84d9bdc0fc7d3c95cc35fa4d5c" exitCode=60 Oct 30 22:00:19.253581 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:19.252934 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-71a3d788-29889960-t7cvk" event={"ID":"3be41157-90f9-4290-b4e3-9db0bf8f5d9c","Type":"ContainerDied","Data":"c28a178840b85be19a4ac6601b1fbb75708d0e84d9bdc0fc7d3c95cc35fa4d5c"} Oct 30 22:00:19.253581 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:19.253278 5748 scope.go:122] "RemoveContainer" containerID="c28a178840b85be19a4ac6601b1fbb75708d0e84d9bdc0fc7d3c95cc35fa4d5c" Oct 30 22:00:19.253581 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:00:19.253531 5748 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-71a3d788-29889960-t7cvk_odh-ai-gateway-infra(3be41157-90f9-4290-b4e3-9db0bf8f5d9c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-71a3d788-29889960-t7cvk" podUID="3be41157-90f9-4290-b4e3-9db0bf8f5d9c" Oct 30 22:00:19.263442 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:19.262960 5748 scope.go:122] "RemoveContainer" containerID="7161b894c7d64d0734bbfca6ed030cf9f8bcd31da3cbedddde8276f67088db2c" Oct 30 22:00:19.263442 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:19.263132 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91874e-29889960-nj2rv" event={"ID":"6990f1c8-d49d-4151-a79a-e231ca0986a4","Type":"ContainerDied","Data":"77d4e5f0b9aa586f2ab22f5d33ca39560bf659051e9b71265761784618d7e6ac"} Oct 30 22:00:19.263442 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:19.263142 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91874e-29889960-nj2rv" Oct 30 22:00:19.267328 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:19.267300 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2c8f56-29889960-m5257" event={"ID":"da3ff95c-c45b-4f37-a080-9ed58a709e1d","Type":"ContainerDied","Data":"e786c9085af175bff687f5810fe50ead2555508f74723882025792b687677c67"} Oct 30 22:00:19.267443 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:19.267395 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2c8f56-29889960-m5257" Oct 30 22:00:19.269978 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:19.269949 5748 generic.go:356] "Generic (PLEG): container finished" podID="a9a3ac51-bd06-4343-b765-5d60748fb62d" containerID="e16940b67f2bee99e9569404851d98cd362c74689917b96f004bb1b6203d9bbf" exitCode=60 Oct 30 22:00:19.270066 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:19.270011 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889960-hm42n" event={"ID":"a9a3ac51-bd06-4343-b765-5d60748fb62d","Type":"ContainerDied","Data":"e16940b67f2bee99e9569404851d98cd362c74689917b96f004bb1b6203d9bbf"} Oct 30 22:00:19.270561 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:19.270541 5748 scope.go:122] "RemoveContainer" containerID="e16940b67f2bee99e9569404851d98cd362c74689917b96f004bb1b6203d9bbf" Oct 30 22:00:19.270856 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:00:19.270833 5748 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889960-hm42n_odh-ai-gateway-infra(a9a3ac51-bd06-4343-b765-5d60748fb62d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889960-hm42n" podUID="a9a3ac51-bd06-4343-b765-5d60748fb62d" Oct 30 22:00:19.280344 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:19.280328 5748 scope.go:122] "RemoveContainer" containerID="6ad30539cd9fc545425e990bc81341e1099e04f3b0dbc4182683f837938c3dbc" Oct 30 22:00:19.310746 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:19.310717 5748 scope.go:122] "RemoveContainer" containerID="982d146746789f68c6b1c874db2ef2a0414ecbb344c82dd5602763da3c5b6f13" Oct 30 22:00:19.322415 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:19.322285 5748 scope.go:122] "RemoveContainer" containerID="5ce3006e5b7ce63d605894940d290ce794fbbe68dfdf089a7264e19b53f568a2" Oct 30 22:00:19.342715 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:19.342695 5748 scope.go:122] "RemoveContainer" containerID="6e8e272f8c1e055abd8c2a799fbcabfc30cfb900a4e51373678b1f2a2c2e503b" Oct 30 22:00:19.350929 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:19.350792 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889960-hm42n"] Oct 30 22:00:19.414587 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:19.414556 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c7d477-29889960-l5cjm" Oct 30 22:00:19.451414 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:19.451389 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ea374-29889960-68g75" Oct 30 22:00:19.470018 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:19.469989 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-71a3d788-29889960-t7cvk"] Oct 30 22:00:19.589816 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:19.589784 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67f2eccf-1409-448c-b69d-eb739ab2195e-kube-api-access-7qpgx\" (UniqueName: \"kubernetes.io/projected/67f2eccf-1409-448c-b69d-eb739ab2195e-kube-api-access-7qpgx\") pod \"67f2eccf-1409-448c-b69d-eb739ab2195e\" (UID: \"67f2eccf-1409-448c-b69d-eb739ab2195e\") " Oct 30 22:00:19.590029 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:19.589841 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67f2eccf-1409-448c-b69d-eb739ab2195e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67f2eccf-1409-448c-b69d-eb739ab2195e-maas-api-service-ca\") pod \"67f2eccf-1409-448c-b69d-eb739ab2195e\" (UID: \"67f2eccf-1409-448c-b69d-eb739ab2195e\") " Oct 30 22:00:19.590029 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:19.589984 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13a18f96-f1ff-436a-8cb9-78d656a45ee4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13a18f96-f1ff-436a-8cb9-78d656a45ee4-maas-api-service-ca\") pod \"13a18f96-f1ff-436a-8cb9-78d656a45ee4\" (UID: \"13a18f96-f1ff-436a-8cb9-78d656a45ee4\") " Oct 30 22:00:19.590029 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:19.590016 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13a18f96-f1ff-436a-8cb9-78d656a45ee4-kube-api-access-pr8hk\" (UniqueName: \"kubernetes.io/projected/13a18f96-f1ff-436a-8cb9-78d656a45ee4-kube-api-access-pr8hk\") pod \"13a18f96-f1ff-436a-8cb9-78d656a45ee4\" (UID: \"13a18f96-f1ff-436a-8cb9-78d656a45ee4\") " Oct 30 22:00:19.590282 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:19.590259 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67f2eccf-1409-448c-b69d-eb739ab2195e-maas-api-service-ca" pod "67f2eccf-1409-448c-b69d-eb739ab2195e" (UID: "67f2eccf-1409-448c-b69d-eb739ab2195e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:19.590345 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:19.590317 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13a18f96-f1ff-436a-8cb9-78d656a45ee4-maas-api-service-ca" pod "13a18f96-f1ff-436a-8cb9-78d656a45ee4" (UID: "13a18f96-f1ff-436a-8cb9-78d656a45ee4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:19.590561 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:19.590539 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13a18f96-f1ff-436a-8cb9-78d656a45ee4-maas-api-service-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:19.590708 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:19.590565 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67f2eccf-1409-448c-b69d-eb739ab2195e-maas-api-service-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:19.592103 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:19.592081 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67f2eccf-1409-448c-b69d-eb739ab2195e-kube-api-access-7qpgx" pod "67f2eccf-1409-448c-b69d-eb739ab2195e" (UID: "67f2eccf-1409-448c-b69d-eb739ab2195e"). InnerVolumeSpecName "kube-api-access-7qpgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:19.592188 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:19.592174 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13a18f96-f1ff-436a-8cb9-78d656a45ee4-kube-api-access-pr8hk" pod "13a18f96-f1ff-436a-8cb9-78d656a45ee4" (UID: "13a18f96-f1ff-436a-8cb9-78d656a45ee4"). InnerVolumeSpecName "kube-api-access-pr8hk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:19.692459 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:19.692431 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qpgx\" (UniqueName: \"kubernetes.io/projected/67f2eccf-1409-448c-b69d-eb739ab2195e-kube-api-access-7qpgx\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:19.692459 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:19.692458 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pr8hk\" (UniqueName: \"kubernetes.io/projected/13a18f96-f1ff-436a-8cb9-78d656a45ee4-kube-api-access-pr8hk\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:19.711944 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:19.711885 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2c8f56-29889960-m5257"] Oct 30 22:00:19.718307 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:19.718284 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2c8f56-29889960-m5257"] Oct 30 22:00:19.966168 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:19.966133 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5"] Oct 30 22:00:19.972239 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:19.972209 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-78c632-openshift-default-6758d5db8c-zftj5"] Oct 30 22:00:19.991044 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:19.991011 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1de75a-29889960-rl48f"] Oct 30 22:00:20.217405 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:20.217321 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91874e-29889960-nj2rv"] Oct 30 22:00:20.223525 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:20.223494 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-91874e-29889960-nj2rv"] Oct 30 22:00:20.282241 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:20.282211 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c7d477-29889960-l5cjm" Oct 30 22:00:20.282436 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:20.282206 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c7d477-29889960-l5cjm" event={"ID":"67f2eccf-1409-448c-b69d-eb739ab2195e","Type":"ContainerDied","Data":"9729627ede3439fcf0f2ae04f5fad89bdb5adc3f4385b4392584baa10c08975e"} Oct 30 22:00:20.282436 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:20.282372 5748 scope.go:122] "RemoveContainer" containerID="0e826933a9d3f1533250091500de93f9ab97339e45e00a3517f79a869b3d47a3" Oct 30 22:00:20.291142 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:20.290723 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ea374-29889960-68g75" event={"ID":"13a18f96-f1ff-436a-8cb9-78d656a45ee4","Type":"ContainerDied","Data":"33d00890901c2643f7d3ac6acbe621553b75e7932c2cee410dafea28a4f0968a"} Oct 30 22:00:20.291142 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:20.290847 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ea374-29889960-68g75" Oct 30 22:00:20.302153 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:20.302128 5748 scope.go:122] "RemoveContainer" containerID="ff9e95ba14eb04859bb299ef5555406f9aa01c676ee9afd443f14d8ae7c3642a" Oct 30 22:00:20.452318 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:20.451324 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1faeab78-65f0-489f-9aeb-d651dd7cac1b" path="/var/lib/kubelet/pods/1faeab78-65f0-489f-9aeb-d651dd7cac1b/volumes" Oct 30 22:00:20.452318 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:20.452146 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6990f1c8-d49d-4151-a79a-e231ca0986a4" path="/var/lib/kubelet/pods/6990f1c8-d49d-4151-a79a-e231ca0986a4/volumes" Oct 30 22:00:20.458082 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:20.453577 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da3ff95c-c45b-4f37-a080-9ed58a709e1d" path="/var/lib/kubelet/pods/da3ff95c-c45b-4f37-a080-9ed58a709e1d/volumes" Oct 30 22:00:20.598440 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:20.598388 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-71a3d788-29889960-t7cvk" Oct 30 22:00:20.619449 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:20.619417 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3be41157-90f9-4290-b4e3-9db0bf8f5d9c-kube-api-access-bvf58\" (UniqueName: \"kubernetes.io/projected/3be41157-90f9-4290-b4e3-9db0bf8f5d9c-kube-api-access-bvf58\") pod \"3be41157-90f9-4290-b4e3-9db0bf8f5d9c\" (UID: \"3be41157-90f9-4290-b4e3-9db0bf8f5d9c\") " Oct 30 22:00:20.619587 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:20.619496 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3be41157-90f9-4290-b4e3-9db0bf8f5d9c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3be41157-90f9-4290-b4e3-9db0bf8f5d9c-maas-api-service-ca\") pod \"3be41157-90f9-4290-b4e3-9db0bf8f5d9c\" (UID: \"3be41157-90f9-4290-b4e3-9db0bf8f5d9c\") " Oct 30 22:00:20.620075 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:20.619964 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3be41157-90f9-4290-b4e3-9db0bf8f5d9c-maas-api-service-ca" pod "3be41157-90f9-4290-b4e3-9db0bf8f5d9c" (UID: "3be41157-90f9-4290-b4e3-9db0bf8f5d9c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:20.620183 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:20.620088 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3be41157-90f9-4290-b4e3-9db0bf8f5d9c-maas-api-service-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:20.622278 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:20.622248 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3be41157-90f9-4290-b4e3-9db0bf8f5d9c-kube-api-access-bvf58" pod "3be41157-90f9-4290-b4e3-9db0bf8f5d9c" (UID: "3be41157-90f9-4290-b4e3-9db0bf8f5d9c"). InnerVolumeSpecName "kube-api-access-bvf58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:20.652239 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:20.652208 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c7d477-29889960-l5cjm"] Oct 30 22:00:20.655191 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:20.655165 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889960-hm42n" Oct 30 22:00:20.656796 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:20.656771 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c7d477-29889960-l5cjm"] Oct 30 22:00:20.659015 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:20.658997 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1de75a-29889960-rl48f" Oct 30 22:00:20.720692 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:20.720667 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f46b5473-f092-4bbb-8486-3c2b1da66231-kube-api-access-qz9rm\" (UniqueName: \"kubernetes.io/projected/f46b5473-f092-4bbb-8486-3c2b1da66231-kube-api-access-qz9rm\") pod \"f46b5473-f092-4bbb-8486-3c2b1da66231\" (UID: \"f46b5473-f092-4bbb-8486-3c2b1da66231\") " Oct 30 22:00:20.720829 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:20.720716 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9a3ac51-bd06-4343-b765-5d60748fb62d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9a3ac51-bd06-4343-b765-5d60748fb62d-maas-api-service-ca\") pod \"a9a3ac51-bd06-4343-b765-5d60748fb62d\" (UID: \"a9a3ac51-bd06-4343-b765-5d60748fb62d\") " Oct 30 22:00:20.720829 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:20.720783 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9a3ac51-bd06-4343-b765-5d60748fb62d-kube-api-access-7j226\" (UniqueName: \"kubernetes.io/projected/a9a3ac51-bd06-4343-b765-5d60748fb62d-kube-api-access-7j226\") pod \"a9a3ac51-bd06-4343-b765-5d60748fb62d\" (UID: \"a9a3ac51-bd06-4343-b765-5d60748fb62d\") " Oct 30 22:00:20.720829 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:20.720823 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f46b5473-f092-4bbb-8486-3c2b1da66231-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f46b5473-f092-4bbb-8486-3c2b1da66231-maas-api-service-ca\") pod \"f46b5473-f092-4bbb-8486-3c2b1da66231\" (UID: \"f46b5473-f092-4bbb-8486-3c2b1da66231\") " Oct 30 22:00:20.721205 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:20.721180 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9a3ac51-bd06-4343-b765-5d60748fb62d-maas-api-service-ca" pod "a9a3ac51-bd06-4343-b765-5d60748fb62d" (UID: "a9a3ac51-bd06-4343-b765-5d60748fb62d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:20.721286 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:20.721179 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f46b5473-f092-4bbb-8486-3c2b1da66231-maas-api-service-ca" pod "f46b5473-f092-4bbb-8486-3c2b1da66231" (UID: "f46b5473-f092-4bbb-8486-3c2b1da66231"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:20.721389 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:20.721374 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f46b5473-f092-4bbb-8486-3c2b1da66231-maas-api-service-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:20.721424 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:20.721391 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9a3ac51-bd06-4343-b765-5d60748fb62d-maas-api-service-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:20.721424 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:20.721400 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvf58\" (UniqueName: \"kubernetes.io/projected/3be41157-90f9-4290-b4e3-9db0bf8f5d9c-kube-api-access-bvf58\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:20.723007 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:20.722984 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f46b5473-f092-4bbb-8486-3c2b1da66231-kube-api-access-qz9rm" pod "f46b5473-f092-4bbb-8486-3c2b1da66231" (UID: "f46b5473-f092-4bbb-8486-3c2b1da66231"). InnerVolumeSpecName "kube-api-access-qz9rm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:20.723174 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:20.723155 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9a3ac51-bd06-4343-b765-5d60748fb62d-kube-api-access-7j226" pod "a9a3ac51-bd06-4343-b765-5d60748fb62d" (UID: "a9a3ac51-bd06-4343-b765-5d60748fb62d"). InnerVolumeSpecName "kube-api-access-7j226". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:20.769331 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:20.769262 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ea374-29889960-68g75"] Oct 30 22:00:20.774417 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:20.774395 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ea374-29889960-68g75"] Oct 30 22:00:20.823518 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:20.823492 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qz9rm\" (UniqueName: \"kubernetes.io/projected/f46b5473-f092-4bbb-8486-3c2b1da66231-kube-api-access-qz9rm\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:20.823518 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:20.823515 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j226\" (UniqueName: \"kubernetes.io/projected/a9a3ac51-bd06-4343-b765-5d60748fb62d-kube-api-access-7j226\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:21.321234 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:21.321197 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1de75a-29889960-rl48f" event={"ID":"f46b5473-f092-4bbb-8486-3c2b1da66231","Type":"ContainerDied","Data":"3830c9f34046cba61c63619f5b8537caa1e0557d33b288dd38b4374bdfa4740a"} Oct 30 22:00:21.321406 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:21.321209 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1de75a-29889960-rl48f" Oct 30 22:00:21.321406 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:21.321264 5748 scope.go:122] "RemoveContainer" containerID="ec34df06e9fceb3cf267e2be50adb14a3b1f5e6439c4c7a53795dc2b56151543" Oct 30 22:00:21.331953 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:21.330862 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-71a3d788-29889960-t7cvk" event={"ID":"3be41157-90f9-4290-b4e3-9db0bf8f5d9c","Type":"ContainerDied","Data":"28a83fca76b32658256da0a92b0775578db21704fb78fadaf7294c792a074ee2"} Oct 30 22:00:21.331953 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:21.331032 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-71a3d788-29889960-t7cvk" Oct 30 22:00:21.338618 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:21.338587 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889960-hm42n" event={"ID":"a9a3ac51-bd06-4343-b765-5d60748fb62d","Type":"ContainerDied","Data":"c759d015017e47c69c805a7ebda118ff0fb94c76045e97beb6acb50654f9ea5b"} Oct 30 22:00:21.338765 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:21.338697 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889960-hm42n" Oct 30 22:00:21.356186 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:21.356084 5748 scope.go:122] "RemoveContainer" containerID="c28a178840b85be19a4ac6601b1fbb75708d0e84d9bdc0fc7d3c95cc35fa4d5c" Oct 30 22:00:21.367577 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:21.367554 5748 scope.go:122] "RemoveContainer" containerID="e16940b67f2bee99e9569404851d98cd362c74689917b96f004bb1b6203d9bbf" Oct 30 22:00:21.659684 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:21.659650 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1de75a-29889960-rl48f"] Oct 30 22:00:21.664309 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:21.664281 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1de75a-29889960-rl48f"] Oct 30 22:00:21.778279 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:21.778245 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889960-hm42n"] Oct 30 22:00:21.784106 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:21.784080 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-69e4b2-29889960-hm42n"] Oct 30 22:00:21.893965 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:21.893935 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-71a3d788-29889960-t7cvk"] Oct 30 22:00:21.901717 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:21.901691 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-71a3d788-29889960-t7cvk"] Oct 30 22:00:22.419234 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:22.419202 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13a18f96-f1ff-436a-8cb9-78d656a45ee4" path="/var/lib/kubelet/pods/13a18f96-f1ff-436a-8cb9-78d656a45ee4/volumes" Oct 30 22:00:22.419692 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:22.419677 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3be41157-90f9-4290-b4e3-9db0bf8f5d9c" path="/var/lib/kubelet/pods/3be41157-90f9-4290-b4e3-9db0bf8f5d9c/volumes" Oct 30 22:00:22.420099 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:22.420084 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67f2eccf-1409-448c-b69d-eb739ab2195e" path="/var/lib/kubelet/pods/67f2eccf-1409-448c-b69d-eb739ab2195e/volumes" Oct 30 22:00:22.420822 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:22.420806 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9a3ac51-bd06-4343-b765-5d60748fb62d" path="/var/lib/kubelet/pods/a9a3ac51-bd06-4343-b765-5d60748fb62d/volumes" Oct 30 22:00:22.421235 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:22.421220 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f46b5473-f092-4bbb-8486-3c2b1da66231" path="/var/lib/kubelet/pods/f46b5473-f092-4bbb-8486-3c2b1da66231/volumes" Oct 30 22:00:23.561613 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.561570 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-ps4v9"] Oct 30 22:00:23.562320 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.562290 5748 cpu_manager.go:414] "containerMap: removing container" podUID="3be41157-90f9-4290-b4e3-9db0bf8f5d9c" containerName="cleanup" Oct 30 22:00:23.562320 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.562316 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3be41157-90f9-4290-b4e3-9db0bf8f5d9c" containerName="cleanup" Oct 30 22:00:23.562511 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.562331 5748 cpu_manager.go:414] "containerMap: removing container" podUID="a9a3ac51-bd06-4343-b765-5d60748fb62d" containerName="cleanup" Oct 30 22:00:23.562511 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.562336 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9a3ac51-bd06-4343-b765-5d60748fb62d" containerName="cleanup" Oct 30 22:00:23.562511 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.562346 5748 cpu_manager.go:414] "containerMap: removing container" podUID="67f2eccf-1409-448c-b69d-eb739ab2195e" containerName="cleanup" Oct 30 22:00:23.562511 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.562350 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67f2eccf-1409-448c-b69d-eb739ab2195e" containerName="cleanup" Oct 30 22:00:23.562511 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.562359 5748 cpu_manager.go:414] "containerMap: removing container" podUID="13a18f96-f1ff-436a-8cb9-78d656a45ee4" containerName="cleanup" Oct 30 22:00:23.562511 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.562364 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13a18f96-f1ff-436a-8cb9-78d656a45ee4" containerName="cleanup" Oct 30 22:00:23.562511 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.562371 5748 cpu_manager.go:414] "containerMap: removing container" podUID="67f2eccf-1409-448c-b69d-eb739ab2195e" containerName="cleanup" Oct 30 22:00:23.562511 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.562376 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67f2eccf-1409-448c-b69d-eb739ab2195e" containerName="cleanup" Oct 30 22:00:23.562511 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.562382 5748 cpu_manager.go:414] "containerMap: removing container" podUID="6990f1c8-d49d-4151-a79a-e231ca0986a4" containerName="cleanup" Oct 30 22:00:23.562511 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.562386 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6990f1c8-d49d-4151-a79a-e231ca0986a4" containerName="cleanup" Oct 30 22:00:23.562511 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.562405 5748 cpu_manager.go:414] "containerMap: removing container" podUID="1faeab78-65f0-489f-9aeb-d651dd7cac1b" containerName="istio-proxy" Oct 30 22:00:23.562511 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.562410 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1faeab78-65f0-489f-9aeb-d651dd7cac1b" containerName="istio-proxy" Oct 30 22:00:23.562511 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.562416 5748 cpu_manager.go:414] "containerMap: removing container" podUID="13a18f96-f1ff-436a-8cb9-78d656a45ee4" containerName="cleanup" Oct 30 22:00:23.562511 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.562421 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13a18f96-f1ff-436a-8cb9-78d656a45ee4" containerName="cleanup" Oct 30 22:00:23.562511 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.562436 5748 cpu_manager.go:414] "containerMap: removing container" podUID="da3ff95c-c45b-4f37-a080-9ed58a709e1d" containerName="cleanup" Oct 30 22:00:23.562511 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.562442 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da3ff95c-c45b-4f37-a080-9ed58a709e1d" containerName="cleanup" Oct 30 22:00:23.562511 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.562450 5748 cpu_manager.go:414] "containerMap: removing container" podUID="3be41157-90f9-4290-b4e3-9db0bf8f5d9c" containerName="cleanup" Oct 30 22:00:23.562511 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.562454 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3be41157-90f9-4290-b4e3-9db0bf8f5d9c" containerName="cleanup" Oct 30 22:00:23.562511 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.562467 5748 cpu_manager.go:414] "containerMap: removing container" podUID="da3ff95c-c45b-4f37-a080-9ed58a709e1d" containerName="cleanup" Oct 30 22:00:23.562511 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.562471 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da3ff95c-c45b-4f37-a080-9ed58a709e1d" containerName="cleanup" Oct 30 22:00:23.562511 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.562484 5748 cpu_manager.go:414] "containerMap: removing container" podUID="1faeab78-65f0-489f-9aeb-d651dd7cac1b" containerName="istio-proxy" Oct 30 22:00:23.562511 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.562489 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1faeab78-65f0-489f-9aeb-d651dd7cac1b" containerName="istio-proxy" Oct 30 22:00:23.562511 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.562498 5748 cpu_manager.go:414] "containerMap: removing container" podUID="e231fcd9-6b46-4f08-bed5-947a82dc6d03" containerName="istio-proxy" Oct 30 22:00:23.562511 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.562503 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e231fcd9-6b46-4f08-bed5-947a82dc6d03" containerName="istio-proxy" Oct 30 22:00:23.562511 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.562509 5748 cpu_manager.go:414] "containerMap: removing container" podUID="da3ff95c-c45b-4f37-a080-9ed58a709e1d" containerName="cleanup" Oct 30 22:00:23.562511 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.562514 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da3ff95c-c45b-4f37-a080-9ed58a709e1d" containerName="cleanup" Oct 30 22:00:23.562511 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.562522 5748 cpu_manager.go:414] "containerMap: removing container" podUID="f46b5473-f092-4bbb-8486-3c2b1da66231" containerName="cleanup" Oct 30 22:00:23.562511 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.562527 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f46b5473-f092-4bbb-8486-3c2b1da66231" containerName="cleanup" Oct 30 22:00:23.563381 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.562534 5748 cpu_manager.go:414] "containerMap: removing container" podUID="f46b5473-f092-4bbb-8486-3c2b1da66231" containerName="cleanup" Oct 30 22:00:23.563381 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.562539 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f46b5473-f092-4bbb-8486-3c2b1da66231" containerName="cleanup" Oct 30 22:00:23.563381 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.562547 5748 cpu_manager.go:414] "containerMap: removing container" podUID="6990f1c8-d49d-4151-a79a-e231ca0986a4" containerName="cleanup" Oct 30 22:00:23.563381 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.562551 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6990f1c8-d49d-4151-a79a-e231ca0986a4" containerName="cleanup" Oct 30 22:00:23.563381 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.562558 5748 cpu_manager.go:414] "containerMap: removing container" podUID="a9a3ac51-bd06-4343-b765-5d60748fb62d" containerName="cleanup" Oct 30 22:00:23.563381 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.562561 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9a3ac51-bd06-4343-b765-5d60748fb62d" containerName="cleanup" Oct 30 22:00:23.563381 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.562568 5748 cpu_manager.go:414] "containerMap: removing container" podUID="f46b5473-f092-4bbb-8486-3c2b1da66231" containerName="cleanup" Oct 30 22:00:23.563381 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.562572 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f46b5473-f092-4bbb-8486-3c2b1da66231" containerName="cleanup" Oct 30 22:00:23.563381 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.562658 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="1faeab78-65f0-489f-9aeb-d651dd7cac1b" containerName="istio-proxy" Oct 30 22:00:23.563381 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.562666 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="13a18f96-f1ff-436a-8cb9-78d656a45ee4" containerName="cleanup" Oct 30 22:00:23.563381 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.562672 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="da3ff95c-c45b-4f37-a080-9ed58a709e1d" containerName="cleanup" Oct 30 22:00:23.563381 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.562677 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="3be41157-90f9-4290-b4e3-9db0bf8f5d9c" containerName="cleanup" Oct 30 22:00:23.563381 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.562686 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="67f2eccf-1409-448c-b69d-eb739ab2195e" containerName="cleanup" Oct 30 22:00:23.563381 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.562690 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="1faeab78-65f0-489f-9aeb-d651dd7cac1b" containerName="istio-proxy" Oct 30 22:00:23.563381 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.562696 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="13a18f96-f1ff-436a-8cb9-78d656a45ee4" containerName="cleanup" Oct 30 22:00:23.563381 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.562702 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="e231fcd9-6b46-4f08-bed5-947a82dc6d03" containerName="istio-proxy" Oct 30 22:00:23.563381 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.562708 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="da3ff95c-c45b-4f37-a080-9ed58a709e1d" containerName="cleanup" Oct 30 22:00:23.563381 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.562712 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="f46b5473-f092-4bbb-8486-3c2b1da66231" containerName="cleanup" Oct 30 22:00:23.563381 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.562717 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="f46b5473-f092-4bbb-8486-3c2b1da66231" containerName="cleanup" Oct 30 22:00:23.563381 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.562722 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="3be41157-90f9-4290-b4e3-9db0bf8f5d9c" containerName="cleanup" Oct 30 22:00:23.563381 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.562727 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="6990f1c8-d49d-4151-a79a-e231ca0986a4" containerName="cleanup" Oct 30 22:00:23.563381 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.562732 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="6990f1c8-d49d-4151-a79a-e231ca0986a4" containerName="cleanup" Oct 30 22:00:23.563381 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.562737 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9a3ac51-bd06-4343-b765-5d60748fb62d" containerName="cleanup" Oct 30 22:00:23.563381 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.562742 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9a3ac51-bd06-4343-b765-5d60748fb62d" containerName="cleanup" Oct 30 22:00:23.563381 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.562747 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="3be41157-90f9-4290-b4e3-9db0bf8f5d9c" containerName="cleanup" Oct 30 22:00:23.563381 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.562752 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9a3ac51-bd06-4343-b765-5d60748fb62d" containerName="cleanup" Oct 30 22:00:23.563381 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.562768 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="67f2eccf-1409-448c-b69d-eb739ab2195e" containerName="cleanup" Oct 30 22:00:23.563381 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.562778 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="67f2eccf-1409-448c-b69d-eb739ab2195e" containerName="cleanup" Oct 30 22:00:23.563381 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.562787 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="6990f1c8-d49d-4151-a79a-e231ca0986a4" containerName="cleanup" Oct 30 22:00:23.574177 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.574153 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-ps4v9" Oct 30 22:00:23.658508 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.658473 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1346c154-7a8b-42a7-96b1-cf9996e38008-istio-data\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-ps4v9\" (UID: \"1346c154-7a8b-42a7-96b1-cf9996e38008\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-ps4v9" Oct 30 22:00:23.658674 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.658534 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1346c154-7a8b-42a7-96b1-cf9996e38008-istio-podinfo\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-ps4v9\" (UID: \"1346c154-7a8b-42a7-96b1-cf9996e38008\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-ps4v9" Oct 30 22:00:23.658674 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.658658 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1346c154-7a8b-42a7-96b1-cf9996e38008-istio-token\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-ps4v9\" (UID: \"1346c154-7a8b-42a7-96b1-cf9996e38008\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-ps4v9" Oct 30 22:00:23.658802 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.658779 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1346c154-7a8b-42a7-96b1-cf9996e38008-credential-socket\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-ps4v9\" (UID: \"1346c154-7a8b-42a7-96b1-cf9996e38008\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-ps4v9" Oct 30 22:00:23.658857 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.658843 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1346c154-7a8b-42a7-96b1-cf9996e38008-istio-envoy\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-ps4v9\" (UID: \"1346c154-7a8b-42a7-96b1-cf9996e38008\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-ps4v9" Oct 30 22:00:23.658945 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.658931 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1346c154-7a8b-42a7-96b1-cf9996e38008-workload-socket\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-ps4v9\" (UID: \"1346c154-7a8b-42a7-96b1-cf9996e38008\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-ps4v9" Oct 30 22:00:23.659060 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.659039 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgs6j\" (UniqueName: \"kubernetes.io/projected/1346c154-7a8b-42a7-96b1-cf9996e38008-kube-api-access-tgs6j\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-ps4v9\" (UID: \"1346c154-7a8b-42a7-96b1-cf9996e38008\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-ps4v9" Oct 30 22:00:23.659132 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.659115 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1346c154-7a8b-42a7-96b1-cf9996e38008-istiod-ca-cert\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-ps4v9\" (UID: \"1346c154-7a8b-42a7-96b1-cf9996e38008\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-ps4v9" Oct 30 22:00:23.659194 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.659180 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1346c154-7a8b-42a7-96b1-cf9996e38008-workload-certs\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-ps4v9\" (UID: \"1346c154-7a8b-42a7-96b1-cf9996e38008\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-ps4v9" Oct 30 22:00:23.708220 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.708193 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-71a3d788-openshift-default-dockercfg-6mkwc\"" Oct 30 22:00:23.760186 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.760156 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1346c154-7a8b-42a7-96b1-cf9996e38008-istio-token\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-ps4v9\" (UID: \"1346c154-7a8b-42a7-96b1-cf9996e38008\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-ps4v9" Oct 30 22:00:23.760315 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.760254 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1346c154-7a8b-42a7-96b1-cf9996e38008-credential-socket\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-ps4v9\" (UID: \"1346c154-7a8b-42a7-96b1-cf9996e38008\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-ps4v9" Oct 30 22:00:23.760315 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.760297 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1346c154-7a8b-42a7-96b1-cf9996e38008-istio-envoy\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-ps4v9\" (UID: \"1346c154-7a8b-42a7-96b1-cf9996e38008\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-ps4v9" Oct 30 22:00:23.760389 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.760365 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1346c154-7a8b-42a7-96b1-cf9996e38008-workload-socket\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-ps4v9\" (UID: \"1346c154-7a8b-42a7-96b1-cf9996e38008\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-ps4v9" Oct 30 22:00:23.760438 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.760401 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgs6j\" (UniqueName: \"kubernetes.io/projected/1346c154-7a8b-42a7-96b1-cf9996e38008-kube-api-access-tgs6j\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-ps4v9\" (UID: \"1346c154-7a8b-42a7-96b1-cf9996e38008\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-ps4v9" Oct 30 22:00:23.760503 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.760479 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1346c154-7a8b-42a7-96b1-cf9996e38008-istiod-ca-cert\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-ps4v9\" (UID: \"1346c154-7a8b-42a7-96b1-cf9996e38008\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-ps4v9" Oct 30 22:00:23.760580 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.760559 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1346c154-7a8b-42a7-96b1-cf9996e38008-workload-certs\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-ps4v9\" (UID: \"1346c154-7a8b-42a7-96b1-cf9996e38008\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-ps4v9" Oct 30 22:00:23.760642 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.760591 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1346c154-7a8b-42a7-96b1-cf9996e38008-istio-data\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-ps4v9\" (UID: \"1346c154-7a8b-42a7-96b1-cf9996e38008\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-ps4v9" Oct 30 22:00:23.760701 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.760655 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1346c154-7a8b-42a7-96b1-cf9996e38008-credential-socket\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-ps4v9\" (UID: \"1346c154-7a8b-42a7-96b1-cf9996e38008\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-ps4v9" Oct 30 22:00:23.760701 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.760677 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1346c154-7a8b-42a7-96b1-cf9996e38008-istio-podinfo\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-ps4v9\" (UID: \"1346c154-7a8b-42a7-96b1-cf9996e38008\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-ps4v9" Oct 30 22:00:23.760806 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.760736 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1346c154-7a8b-42a7-96b1-cf9996e38008-workload-socket\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-ps4v9\" (UID: \"1346c154-7a8b-42a7-96b1-cf9996e38008\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-ps4v9" Oct 30 22:00:23.761167 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.761139 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1346c154-7a8b-42a7-96b1-cf9996e38008-workload-certs\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-ps4v9\" (UID: \"1346c154-7a8b-42a7-96b1-cf9996e38008\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-ps4v9" Oct 30 22:00:23.761277 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.761174 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1346c154-7a8b-42a7-96b1-cf9996e38008-istio-data\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-ps4v9\" (UID: \"1346c154-7a8b-42a7-96b1-cf9996e38008\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-ps4v9" Oct 30 22:00:23.761277 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.761231 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1346c154-7a8b-42a7-96b1-cf9996e38008-istiod-ca-cert\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-ps4v9\" (UID: \"1346c154-7a8b-42a7-96b1-cf9996e38008\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-ps4v9" Oct 30 22:00:23.762935 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.762915 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1346c154-7a8b-42a7-96b1-cf9996e38008-istio-envoy\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-ps4v9\" (UID: \"1346c154-7a8b-42a7-96b1-cf9996e38008\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-ps4v9" Oct 30 22:00:23.763176 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.763156 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1346c154-7a8b-42a7-96b1-cf9996e38008-istio-podinfo\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-ps4v9\" (UID: \"1346c154-7a8b-42a7-96b1-cf9996e38008\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-ps4v9" Oct 30 22:00:23.789251 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.788624 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-ps4v9"] Oct 30 22:00:23.822407 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.822347 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgs6j\" (UniqueName: \"kubernetes.io/projected/1346c154-7a8b-42a7-96b1-cf9996e38008-kube-api-access-tgs6j\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-ps4v9\" (UID: \"1346c154-7a8b-42a7-96b1-cf9996e38008\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-ps4v9" Oct 30 22:00:23.822514 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.822402 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1346c154-7a8b-42a7-96b1-cf9996e38008-istio-token\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-ps4v9\" (UID: \"1346c154-7a8b-42a7-96b1-cf9996e38008\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-ps4v9" Oct 30 22:00:23.889573 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:23.889539 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-ps4v9" Oct 30 22:00:24.327351 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:24.327324 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-ps4v9"] Oct 30 22:00:24.329282 ip-10-0-143-61 kubenswrapper[5748]: W1030 22:00:24.329234 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1346c154_7a8b_42a7_96b1_cf9996e38008.slice/crio-7f6adb530241e3d292b6370d8374f55b54945581bdf3be02ac7a175747f21a99 WatchSource:0}: Error finding container 7f6adb530241e3d292b6370d8374f55b54945581bdf3be02ac7a175747f21a99: Status 404 returned error can't find the container with id 7f6adb530241e3d292b6370d8374f55b54945581bdf3be02ac7a175747f21a99 Oct 30 22:00:24.332166 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:24.332131 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:00:24.332239 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:24.332199 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:00:24.332239 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:24.332227 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:00:24.367223 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:24.367200 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-ps4v9" event={"ID":"1346c154-7a8b-42a7-96b1-cf9996e38008","Type":"ContainerStarted","Data":"7f6adb530241e3d292b6370d8374f55b54945581bdf3be02ac7a175747f21a99"} Oct 30 22:00:25.421351 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:25.421308 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-ps4v9" event={"ID":"1346c154-7a8b-42a7-96b1-cf9996e38008","Type":"ContainerStarted","Data":"8cb3695a13e4ebdeebf218c3794dcbeb97513ef215a55a9c1414662a6b29c1f7"} Oct 30 22:00:25.627353 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:25.627295 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-ps4v9" podStartSLOduration=2.627281234 podStartE2EDuration="2.627281234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:25.625254229 +0000 UTC m=+2209.367775166" watchObservedRunningTime="2026-10-30 22:00:25.627281234 +0000 UTC m=+2209.369802169" Oct 30 22:00:25.892929 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:25.890513 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-ps4v9" Oct 30 22:00:25.892929 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:25.892423 5748 patch_prober.go:29] interesting pod/e2e-mt-71a3d788-openshift-default-778558588f-ps4v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 30 22:00:25.892929 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:25.892482 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-ps4v9" podUID="1346c154-7a8b-42a7-96b1-cf9996e38008" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 30 22:00:26.890031 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:26.889995 5748 patch_prober.go:29] interesting pod/e2e-mt-71a3d788-openshift-default-778558588f-ps4v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 30 22:00:26.890436 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:26.890045 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-ps4v9" podUID="1346c154-7a8b-42a7-96b1-cf9996e38008" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 30 22:00:27.890604 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:27.890568 5748 patch_prober.go:29] interesting pod/e2e-mt-71a3d788-openshift-default-778558588f-ps4v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 30 22:00:27.891011 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:27.890619 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-ps4v9" podUID="1346c154-7a8b-42a7-96b1-cf9996e38008" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 30 22:00:28.890768 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:28.890732 5748 patch_prober.go:29] interesting pod/e2e-mt-71a3d788-openshift-default-778558588f-ps4v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 30 22:00:28.891200 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:28.890787 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-ps4v9" podUID="1346c154-7a8b-42a7-96b1-cf9996e38008" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 30 22:00:29.890936 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:29.890891 5748 patch_prober.go:29] interesting pod/e2e-mt-71a3d788-openshift-default-778558588f-ps4v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 30 22:00:29.891419 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:29.890964 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-ps4v9" podUID="1346c154-7a8b-42a7-96b1-cf9996e38008" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 30 22:00:30.152551 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:00:30.152472 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:00:31.026453 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:31.026400 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-ps4v9" Oct 30 22:00:31.026975 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:31.026944 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-ps4v9" Oct 30 22:00:31.027843 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:31.027816 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-ps4v9" Oct 30 22:00:38.691218 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:38.691186 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-cx2t5"] Oct 30 22:00:38.691742 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:38.691726 5748 cpu_manager.go:414] "containerMap: removing container" podUID="13a18f96-f1ff-436a-8cb9-78d656a45ee4" containerName="cleanup" Oct 30 22:00:38.691787 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:38.691743 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13a18f96-f1ff-436a-8cb9-78d656a45ee4" containerName="cleanup" Oct 30 22:00:38.691787 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:38.691780 5748 cpu_manager.go:414] "containerMap: removing container" podUID="3be41157-90f9-4290-b4e3-9db0bf8f5d9c" containerName="cleanup" Oct 30 22:00:38.691787 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:38.691785 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3be41157-90f9-4290-b4e3-9db0bf8f5d9c" containerName="cleanup" Oct 30 22:00:38.691885 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:38.691799 5748 cpu_manager.go:414] "containerMap: removing container" podUID="a9a3ac51-bd06-4343-b765-5d60748fb62d" containerName="cleanup" Oct 30 22:00:38.691885 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:38.691807 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9a3ac51-bd06-4343-b765-5d60748fb62d" containerName="cleanup" Oct 30 22:00:38.691885 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:38.691822 5748 cpu_manager.go:414] "containerMap: removing container" podUID="67f2eccf-1409-448c-b69d-eb739ab2195e" containerName="cleanup" Oct 30 22:00:38.691885 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:38.691828 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67f2eccf-1409-448c-b69d-eb739ab2195e" containerName="cleanup" Oct 30 22:00:38.691885 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:38.691834 5748 cpu_manager.go:414] "containerMap: removing container" podUID="6990f1c8-d49d-4151-a79a-e231ca0986a4" containerName="cleanup" Oct 30 22:00:38.691885 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:38.691840 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6990f1c8-d49d-4151-a79a-e231ca0986a4" containerName="cleanup" Oct 30 22:00:38.692090 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:38.691945 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="da3ff95c-c45b-4f37-a080-9ed58a709e1d" containerName="cleanup" Oct 30 22:00:38.692090 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:38.691963 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="13a18f96-f1ff-436a-8cb9-78d656a45ee4" containerName="cleanup" Oct 30 22:00:38.692090 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:38.691972 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="f46b5473-f092-4bbb-8486-3c2b1da66231" containerName="cleanup" Oct 30 22:00:38.697073 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:38.697053 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-cx2t5" Oct 30 22:00:38.783939 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:38.779070 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53df3a33-7883-4793-b495-e5c7543432f0-credential-socket\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-cx2t5\" (UID: \"53df3a33-7883-4793-b495-e5c7543432f0\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-cx2t5" Oct 30 22:00:38.783939 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:38.779131 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53df3a33-7883-4793-b495-e5c7543432f0-istio-envoy\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-cx2t5\" (UID: \"53df3a33-7883-4793-b495-e5c7543432f0\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-cx2t5" Oct 30 22:00:38.783939 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:38.779269 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53df3a33-7883-4793-b495-e5c7543432f0-workload-socket\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-cx2t5\" (UID: \"53df3a33-7883-4793-b495-e5c7543432f0\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-cx2t5" Oct 30 22:00:38.783939 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:38.779303 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53df3a33-7883-4793-b495-e5c7543432f0-istiod-ca-cert\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-cx2t5\" (UID: \"53df3a33-7883-4793-b495-e5c7543432f0\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-cx2t5" Oct 30 22:00:38.783939 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:38.779401 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53df3a33-7883-4793-b495-e5c7543432f0-istio-data\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-cx2t5\" (UID: \"53df3a33-7883-4793-b495-e5c7543432f0\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-cx2t5" Oct 30 22:00:38.783939 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:38.779430 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt7g7\" (UniqueName: \"kubernetes.io/projected/53df3a33-7883-4793-b495-e5c7543432f0-kube-api-access-gt7g7\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-cx2t5\" (UID: \"53df3a33-7883-4793-b495-e5c7543432f0\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-cx2t5" Oct 30 22:00:38.783939 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:38.779509 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53df3a33-7883-4793-b495-e5c7543432f0-istio-podinfo\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-cx2t5\" (UID: \"53df3a33-7883-4793-b495-e5c7543432f0\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-cx2t5" Oct 30 22:00:38.783939 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:38.779565 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53df3a33-7883-4793-b495-e5c7543432f0-workload-certs\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-cx2t5\" (UID: \"53df3a33-7883-4793-b495-e5c7543432f0\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-cx2t5" Oct 30 22:00:38.783939 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:38.779613 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53df3a33-7883-4793-b495-e5c7543432f0-istio-token\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-cx2t5\" (UID: \"53df3a33-7883-4793-b495-e5c7543432f0\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-cx2t5" Oct 30 22:00:38.803182 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:38.803152 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-cx2t5"] Oct 30 22:00:38.880752 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:38.880720 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53df3a33-7883-4793-b495-e5c7543432f0-istio-token\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-cx2t5\" (UID: \"53df3a33-7883-4793-b495-e5c7543432f0\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-cx2t5" Oct 30 22:00:38.880940 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:38.880760 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53df3a33-7883-4793-b495-e5c7543432f0-credential-socket\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-cx2t5\" (UID: \"53df3a33-7883-4793-b495-e5c7543432f0\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-cx2t5" Oct 30 22:00:38.880940 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:38.880783 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53df3a33-7883-4793-b495-e5c7543432f0-istio-envoy\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-cx2t5\" (UID: \"53df3a33-7883-4793-b495-e5c7543432f0\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-cx2t5" Oct 30 22:00:38.880940 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:38.880845 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53df3a33-7883-4793-b495-e5c7543432f0-workload-socket\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-cx2t5\" (UID: \"53df3a33-7883-4793-b495-e5c7543432f0\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-cx2t5" Oct 30 22:00:38.880940 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:38.880872 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53df3a33-7883-4793-b495-e5c7543432f0-istiod-ca-cert\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-cx2t5\" (UID: \"53df3a33-7883-4793-b495-e5c7543432f0\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-cx2t5" Oct 30 22:00:38.881176 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:38.880980 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53df3a33-7883-4793-b495-e5c7543432f0-istio-data\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-cx2t5\" (UID: \"53df3a33-7883-4793-b495-e5c7543432f0\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-cx2t5" Oct 30 22:00:38.881176 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:38.881010 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt7g7\" (UniqueName: \"kubernetes.io/projected/53df3a33-7883-4793-b495-e5c7543432f0-kube-api-access-gt7g7\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-cx2t5\" (UID: \"53df3a33-7883-4793-b495-e5c7543432f0\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-cx2t5" Oct 30 22:00:38.881176 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:38.881119 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53df3a33-7883-4793-b495-e5c7543432f0-istio-podinfo\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-cx2t5\" (UID: \"53df3a33-7883-4793-b495-e5c7543432f0\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-cx2t5" Oct 30 22:00:38.881176 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:38.881172 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53df3a33-7883-4793-b495-e5c7543432f0-workload-certs\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-cx2t5\" (UID: \"53df3a33-7883-4793-b495-e5c7543432f0\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-cx2t5" Oct 30 22:00:38.881400 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:38.881242 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53df3a33-7883-4793-b495-e5c7543432f0-credential-socket\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-cx2t5\" (UID: \"53df3a33-7883-4793-b495-e5c7543432f0\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-cx2t5" Oct 30 22:00:38.881400 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:38.881270 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53df3a33-7883-4793-b495-e5c7543432f0-workload-socket\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-cx2t5\" (UID: \"53df3a33-7883-4793-b495-e5c7543432f0\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-cx2t5" Oct 30 22:00:38.881400 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:38.881323 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53df3a33-7883-4793-b495-e5c7543432f0-istio-data\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-cx2t5\" (UID: \"53df3a33-7883-4793-b495-e5c7543432f0\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-cx2t5" Oct 30 22:00:38.881575 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:38.881512 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53df3a33-7883-4793-b495-e5c7543432f0-workload-certs\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-cx2t5\" (UID: \"53df3a33-7883-4793-b495-e5c7543432f0\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-cx2t5" Oct 30 22:00:38.881743 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:38.881721 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53df3a33-7883-4793-b495-e5c7543432f0-istiod-ca-cert\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-cx2t5\" (UID: \"53df3a33-7883-4793-b495-e5c7543432f0\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-cx2t5" Oct 30 22:00:38.883137 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:38.883116 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53df3a33-7883-4793-b495-e5c7543432f0-istio-envoy\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-cx2t5\" (UID: \"53df3a33-7883-4793-b495-e5c7543432f0\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-cx2t5" Oct 30 22:00:38.883458 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:38.883440 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53df3a33-7883-4793-b495-e5c7543432f0-istio-podinfo\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-cx2t5\" (UID: \"53df3a33-7883-4793-b495-e5c7543432f0\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-cx2t5" Oct 30 22:00:38.945731 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:38.945677 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53df3a33-7883-4793-b495-e5c7543432f0-istio-token\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-cx2t5\" (UID: \"53df3a33-7883-4793-b495-e5c7543432f0\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-cx2t5" Oct 30 22:00:38.945886 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:38.945867 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt7g7\" (UniqueName: \"kubernetes.io/projected/53df3a33-7883-4793-b495-e5c7543432f0-kube-api-access-gt7g7\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-cx2t5\" (UID: \"53df3a33-7883-4793-b495-e5c7543432f0\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-cx2t5" Oct 30 22:00:39.010787 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:39.010760 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-cx2t5" Oct 30 22:00:39.441575 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:39.441543 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-cx2t5"] Oct 30 22:00:39.442744 ip-10-0-143-61 kubenswrapper[5748]: W1030 22:00:39.442711 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53df3a33_7883_4793_b495_e5c7543432f0.slice/crio-eb5cd5e03a9aa92e1e171ea7e4105632e2cb30de312f82b7c84e216e0cb4c4e1 WatchSource:0}: Error finding container eb5cd5e03a9aa92e1e171ea7e4105632e2cb30de312f82b7c84e216e0cb4c4e1: Status 404 returned error can't find the container with id eb5cd5e03a9aa92e1e171ea7e4105632e2cb30de312f82b7c84e216e0cb4c4e1 Oct 30 22:00:39.445181 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:39.445146 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:00:39.445260 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:39.445211 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:00:39.445260 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:39.445240 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:00:39.520913 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:39.520868 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-cx2t5" event={"ID":"53df3a33-7883-4793-b495-e5c7543432f0","Type":"ContainerStarted","Data":"60220c4b1466575ccac22c1104f98508c572300d735da23cd540b5297a3a8211"} Oct 30 22:00:39.521027 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:39.520944 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-cx2t5" event={"ID":"53df3a33-7883-4793-b495-e5c7543432f0","Type":"ContainerStarted","Data":"eb5cd5e03a9aa92e1e171ea7e4105632e2cb30de312f82b7c84e216e0cb4c4e1"} Oct 30 22:00:39.644494 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:39.644431 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-cx2t5" podStartSLOduration=1.644415942 podStartE2EDuration="1.644415942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:39.642242687 +0000 UTC m=+2223.384763623" watchObservedRunningTime="2026-10-30 22:00:39.644415942 +0000 UTC m=+2223.386936877" Oct 30 22:00:40.015744 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:40.012212 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-cx2t5" Oct 30 22:00:40.015744 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:40.015576 5748 patch_prober.go:29] interesting pod/e2e-mt-71a3d788-openshift-default-778558588f-cx2t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 30 22:00:40.016449 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:40.015669 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-cx2t5" podUID="53df3a33-7883-4793-b495-e5c7543432f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 30 22:00:41.012317 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:41.012280 5748 patch_prober.go:29] interesting pod/e2e-mt-71a3d788-openshift-default-778558588f-cx2t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 30 22:00:41.012570 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:41.012542 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-cx2t5" podUID="53df3a33-7883-4793-b495-e5c7543432f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 30 22:00:42.015920 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:42.015440 5748 patch_prober.go:29] interesting pod/e2e-mt-71a3d788-openshift-default-778558588f-cx2t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 30 22:00:42.015920 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:42.015520 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-cx2t5" podUID="53df3a33-7883-4793-b495-e5c7543432f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 30 22:00:43.011921 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:43.011332 5748 patch_prober.go:29] interesting pod/e2e-mt-71a3d788-openshift-default-778558588f-cx2t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 30 22:00:43.011921 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:43.011399 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-cx2t5" podUID="53df3a33-7883-4793-b495-e5c7543432f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 30 22:00:44.018922 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:44.018271 5748 patch_prober.go:29] interesting pod/e2e-mt-71a3d788-openshift-default-778558588f-cx2t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 30 22:00:44.018922 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:44.018332 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-cx2t5" podUID="53df3a33-7883-4793-b495-e5c7543432f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 30 22:00:45.021294 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:45.021253 5748 patch_prober.go:29] interesting pod/e2e-mt-71a3d788-openshift-default-778558588f-cx2t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 30 22:00:45.021865 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:45.021830 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-cx2t5" podUID="53df3a33-7883-4793-b495-e5c7543432f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 30 22:00:46.334764 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:46.334730 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-cx2t5" Oct 30 22:00:46.335248 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:46.335220 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-cx2t5" Oct 30 22:00:46.336500 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:46.336473 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-cx2t5" Oct 30 22:00:53.808661 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:53.808624 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-6vzg4"] Oct 30 22:00:53.813093 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:53.813011 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-6vzg4" Oct 30 22:00:53.912419 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:53.912389 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-6vzg4"] Oct 30 22:00:53.928358 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:53.928329 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e9ffa82-546b-44a0-866b-44c89dfbb43e-credential-socket\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-6vzg4\" (UID: \"0e9ffa82-546b-44a0-866b-44c89dfbb43e\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-6vzg4" Oct 30 22:00:53.928504 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:53.928373 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e9ffa82-546b-44a0-866b-44c89dfbb43e-istiod-ca-cert\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-6vzg4\" (UID: \"0e9ffa82-546b-44a0-866b-44c89dfbb43e\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-6vzg4" Oct 30 22:00:53.928504 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:53.928482 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjvtd\" (UniqueName: \"kubernetes.io/projected/0e9ffa82-546b-44a0-866b-44c89dfbb43e-kube-api-access-pjvtd\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-6vzg4\" (UID: \"0e9ffa82-546b-44a0-866b-44c89dfbb43e\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-6vzg4" Oct 30 22:00:53.928635 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:53.928543 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e9ffa82-546b-44a0-866b-44c89dfbb43e-istio-data\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-6vzg4\" (UID: \"0e9ffa82-546b-44a0-866b-44c89dfbb43e\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-6vzg4" Oct 30 22:00:53.928697 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:53.928645 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e9ffa82-546b-44a0-866b-44c89dfbb43e-istio-podinfo\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-6vzg4\" (UID: \"0e9ffa82-546b-44a0-866b-44c89dfbb43e\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-6vzg4" Oct 30 22:00:53.928753 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:53.928704 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e9ffa82-546b-44a0-866b-44c89dfbb43e-istio-token\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-6vzg4\" (UID: \"0e9ffa82-546b-44a0-866b-44c89dfbb43e\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-6vzg4" Oct 30 22:00:53.928753 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:53.928745 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e9ffa82-546b-44a0-866b-44c89dfbb43e-workload-socket\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-6vzg4\" (UID: \"0e9ffa82-546b-44a0-866b-44c89dfbb43e\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-6vzg4" Oct 30 22:00:53.928857 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:53.928787 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e9ffa82-546b-44a0-866b-44c89dfbb43e-istio-envoy\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-6vzg4\" (UID: \"0e9ffa82-546b-44a0-866b-44c89dfbb43e\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-6vzg4" Oct 30 22:00:53.928857 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:53.928805 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e9ffa82-546b-44a0-866b-44c89dfbb43e-workload-certs\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-6vzg4\" (UID: \"0e9ffa82-546b-44a0-866b-44c89dfbb43e\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-6vzg4" Oct 30 22:00:54.030199 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:54.030164 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e9ffa82-546b-44a0-866b-44c89dfbb43e-istio-podinfo\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-6vzg4\" (UID: \"0e9ffa82-546b-44a0-866b-44c89dfbb43e\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-6vzg4" Oct 30 22:00:54.030362 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:54.030207 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e9ffa82-546b-44a0-866b-44c89dfbb43e-istio-token\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-6vzg4\" (UID: \"0e9ffa82-546b-44a0-866b-44c89dfbb43e\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-6vzg4" Oct 30 22:00:54.030362 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:54.030231 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e9ffa82-546b-44a0-866b-44c89dfbb43e-workload-socket\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-6vzg4\" (UID: \"0e9ffa82-546b-44a0-866b-44c89dfbb43e\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-6vzg4" Oct 30 22:00:54.030362 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:54.030269 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e9ffa82-546b-44a0-866b-44c89dfbb43e-istio-envoy\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-6vzg4\" (UID: \"0e9ffa82-546b-44a0-866b-44c89dfbb43e\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-6vzg4" Oct 30 22:00:54.030362 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:54.030288 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e9ffa82-546b-44a0-866b-44c89dfbb43e-workload-certs\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-6vzg4\" (UID: \"0e9ffa82-546b-44a0-866b-44c89dfbb43e\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-6vzg4" Oct 30 22:00:54.030362 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:54.030330 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e9ffa82-546b-44a0-866b-44c89dfbb43e-credential-socket\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-6vzg4\" (UID: \"0e9ffa82-546b-44a0-866b-44c89dfbb43e\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-6vzg4" Oct 30 22:00:54.030362 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:54.030357 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e9ffa82-546b-44a0-866b-44c89dfbb43e-istiod-ca-cert\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-6vzg4\" (UID: \"0e9ffa82-546b-44a0-866b-44c89dfbb43e\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-6vzg4" Oct 30 22:00:54.030688 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:54.030402 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjvtd\" (UniqueName: \"kubernetes.io/projected/0e9ffa82-546b-44a0-866b-44c89dfbb43e-kube-api-access-pjvtd\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-6vzg4\" (UID: \"0e9ffa82-546b-44a0-866b-44c89dfbb43e\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-6vzg4" Oct 30 22:00:54.030688 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:54.030478 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e9ffa82-546b-44a0-866b-44c89dfbb43e-istio-data\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-6vzg4\" (UID: \"0e9ffa82-546b-44a0-866b-44c89dfbb43e\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-6vzg4" Oct 30 22:00:54.030810 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:54.030693 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e9ffa82-546b-44a0-866b-44c89dfbb43e-workload-socket\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-6vzg4\" (UID: \"0e9ffa82-546b-44a0-866b-44c89dfbb43e\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-6vzg4" Oct 30 22:00:54.030810 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:54.030739 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e9ffa82-546b-44a0-866b-44c89dfbb43e-workload-certs\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-6vzg4\" (UID: \"0e9ffa82-546b-44a0-866b-44c89dfbb43e\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-6vzg4" Oct 30 22:00:54.030952 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:54.030886 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e9ffa82-546b-44a0-866b-44c89dfbb43e-credential-socket\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-6vzg4\" (UID: \"0e9ffa82-546b-44a0-866b-44c89dfbb43e\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-6vzg4" Oct 30 22:00:54.031032 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:54.030981 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e9ffa82-546b-44a0-866b-44c89dfbb43e-istio-data\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-6vzg4\" (UID: \"0e9ffa82-546b-44a0-866b-44c89dfbb43e\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-6vzg4" Oct 30 22:00:54.031139 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:54.031115 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e9ffa82-546b-44a0-866b-44c89dfbb43e-istiod-ca-cert\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-6vzg4\" (UID: \"0e9ffa82-546b-44a0-866b-44c89dfbb43e\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-6vzg4" Oct 30 22:00:54.032610 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:54.032589 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e9ffa82-546b-44a0-866b-44c89dfbb43e-istio-envoy\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-6vzg4\" (UID: \"0e9ffa82-546b-44a0-866b-44c89dfbb43e\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-6vzg4" Oct 30 22:00:54.032756 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:54.032739 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e9ffa82-546b-44a0-866b-44c89dfbb43e-istio-podinfo\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-6vzg4\" (UID: \"0e9ffa82-546b-44a0-866b-44c89dfbb43e\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-6vzg4" Oct 30 22:00:54.092410 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:54.092349 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e9ffa82-546b-44a0-866b-44c89dfbb43e-istio-token\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-6vzg4\" (UID: \"0e9ffa82-546b-44a0-866b-44c89dfbb43e\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-6vzg4" Oct 30 22:00:54.092520 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:54.092457 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjvtd\" (UniqueName: \"kubernetes.io/projected/0e9ffa82-546b-44a0-866b-44c89dfbb43e-kube-api-access-pjvtd\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-6vzg4\" (UID: \"0e9ffa82-546b-44a0-866b-44c89dfbb43e\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-6vzg4" Oct 30 22:00:54.127885 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:54.127855 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-6vzg4" Oct 30 22:00:54.561077 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:54.561055 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-6vzg4"] Oct 30 22:00:54.563111 ip-10-0-143-61 kubenswrapper[5748]: W1030 22:00:54.563081 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e9ffa82_546b_44a0_866b_44c89dfbb43e.slice/crio-5e14cadecb4d189ef4d14834a35eeed77aba063ec14af4ab555448243259daad WatchSource:0}: Error finding container 5e14cadecb4d189ef4d14834a35eeed77aba063ec14af4ab555448243259daad: Status 404 returned error can't find the container with id 5e14cadecb4d189ef4d14834a35eeed77aba063ec14af4ab555448243259daad Oct 30 22:00:54.565566 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:54.565532 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:00:54.565664 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:54.565596 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:00:54.565664 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:54.565625 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:00:54.709561 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:54.709457 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-6vzg4" event={"ID":"0e9ffa82-546b-44a0-866b-44c89dfbb43e","Type":"ContainerStarted","Data":"1ddc6a01a5dc83bb1a2ff01f7c9f86bff070bc6be3f497c234534384fa0f0ab1"} Oct 30 22:00:54.709561 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:54.709532 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-6vzg4" event={"ID":"0e9ffa82-546b-44a0-866b-44c89dfbb43e","Type":"ContainerStarted","Data":"5e14cadecb4d189ef4d14834a35eeed77aba063ec14af4ab555448243259daad"} Oct 30 22:00:54.833378 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:54.833325 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-6vzg4" podStartSLOduration=1.833307451 podStartE2EDuration="1.833307451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:54.831189396 +0000 UTC m=+2238.573710339" watchObservedRunningTime="2026-10-30 22:00:54.833307451 +0000 UTC m=+2238.575828386" Oct 30 22:00:55.128388 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:55.128309 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-6vzg4" Oct 30 22:00:55.129810 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:55.129779 5748 patch_prober.go:29] interesting pod/e2e-mt-71a3d788-openshift-default-778558588f-6vzg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 30 22:00:55.129968 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:55.129841 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-6vzg4" podUID="0e9ffa82-546b-44a0-866b-44c89dfbb43e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 30 22:00:56.128444 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:56.128403 5748 patch_prober.go:29] interesting pod/e2e-mt-71a3d788-openshift-default-778558588f-6vzg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 30 22:00:56.128875 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:56.128465 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-6vzg4" podUID="0e9ffa82-546b-44a0-866b-44c89dfbb43e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 30 22:00:57.129078 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:57.129040 5748 patch_prober.go:29] interesting pod/e2e-mt-71a3d788-openshift-default-778558588f-6vzg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 30 22:00:57.129453 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:57.129097 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-6vzg4" podUID="0e9ffa82-546b-44a0-866b-44c89dfbb43e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 30 22:00:58.128413 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:58.128377 5748 patch_prober.go:29] interesting pod/e2e-mt-71a3d788-openshift-default-778558588f-6vzg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 30 22:00:58.128609 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:58.128433 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-6vzg4" podUID="0e9ffa82-546b-44a0-866b-44c89dfbb43e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 30 22:00:59.128500 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:59.128465 5748 patch_prober.go:29] interesting pod/e2e-mt-71a3d788-openshift-default-778558588f-6vzg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 30 22:00:59.129127 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:00:59.129098 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-6vzg4" podUID="0e9ffa82-546b-44a0-866b-44c89dfbb43e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 30 22:01:00.096066 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:01:00.096029 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71a3d788-tb49x"] Oct 30 22:01:00.112930 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:01:00.112146 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71a3d788-tb49x" Oct 30 22:01:00.129125 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:01:00.129051 5748 patch_prober.go:29] interesting pod/e2e-mt-71a3d788-openshift-default-778558588f-6vzg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 30 22:01:00.129125 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:01:00.129107 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-6vzg4" podUID="0e9ffa82-546b-44a0-866b-44c89dfbb43e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 30 22:01:00.164948 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:01:00.164889 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71a3d788-tb49x"] Oct 30 22:01:00.172928 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:01:00.170477 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:01:00.227296 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:01:00.227259 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 22:01:00.227871 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:01:00.227598 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:01:00.228049 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:01:00.227701 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:01:00.228147 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:01:00.227803 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:01:00.264208 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:01:00.263447 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw6kl\" (UniqueName: \"kubernetes.io/projected/8e64a170-0a90-4e0b-b4ce-87e92d676ac1-kube-api-access-tw6kl\") pod \"maas-api-revoke-keys-e2e-mt-71a3d788-tb49x\" (UID: \"8e64a170-0a90-4e0b-b4ce-87e92d676ac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71a3d788-tb49x" Oct 30 22:01:00.264208 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:01:00.263615 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e64a170-0a90-4e0b-b4ce-87e92d676ac1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-71a3d788-tb49x\" (UID: \"8e64a170-0a90-4e0b-b4ce-87e92d676ac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71a3d788-tb49x" Oct 30 22:01:00.264208 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:01:00.263648 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e64a170-0a90-4e0b-b4ce-87e92d676ac1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-71a3d788-tb49x\" (UID: \"8e64a170-0a90-4e0b-b4ce-87e92d676ac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71a3d788-tb49x" Oct 30 22:01:00.364860 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:01:00.364766 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw6kl\" (UniqueName: \"kubernetes.io/projected/8e64a170-0a90-4e0b-b4ce-87e92d676ac1-kube-api-access-tw6kl\") pod \"maas-api-revoke-keys-e2e-mt-71a3d788-tb49x\" (UID: \"8e64a170-0a90-4e0b-b4ce-87e92d676ac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71a3d788-tb49x" Oct 30 22:01:00.365043 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:01:00.364890 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e64a170-0a90-4e0b-b4ce-87e92d676ac1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-71a3d788-tb49x\" (UID: \"8e64a170-0a90-4e0b-b4ce-87e92d676ac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71a3d788-tb49x" Oct 30 22:01:00.365043 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:01:00.364931 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e64a170-0a90-4e0b-b4ce-87e92d676ac1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-71a3d788-tb49x\" (UID: \"8e64a170-0a90-4e0b-b4ce-87e92d676ac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71a3d788-tb49x" Oct 30 22:01:00.365623 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:01:00.365600 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e64a170-0a90-4e0b-b4ce-87e92d676ac1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-71a3d788-tb49x\" (UID: \"8e64a170-0a90-4e0b-b4ce-87e92d676ac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71a3d788-tb49x" Oct 30 22:01:00.368208 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:01:00.367888 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e64a170-0a90-4e0b-b4ce-87e92d676ac1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-71a3d788-tb49x\" (UID: \"8e64a170-0a90-4e0b-b4ce-87e92d676ac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71a3d788-tb49x" Oct 30 22:01:00.428625 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:01:00.428593 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw6kl\" (UniqueName: \"kubernetes.io/projected/8e64a170-0a90-4e0b-b4ce-87e92d676ac1-kube-api-access-tw6kl\") pod \"maas-api-revoke-keys-e2e-mt-71a3d788-tb49x\" (UID: \"8e64a170-0a90-4e0b-b4ce-87e92d676ac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71a3d788-tb49x" Oct 30 22:01:00.442561 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:01:00.442533 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71a3d788-tb49x" Oct 30 22:01:00.822365 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:01:00.821984 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71a3d788-tb49x"] Oct 30 22:01:00.825382 ip-10-0-143-61 kubenswrapper[5748]: W1030 22:01:00.825345 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e64a170_0a90_4e0b_b4ce_87e92d676ac1.slice/crio-75c58b355d0dcaba28c86365370b8d40391da1267f550ba8668fa6ee237367b6 WatchSource:0}: Error finding container 75c58b355d0dcaba28c86365370b8d40391da1267f550ba8668fa6ee237367b6: Status 404 returned error can't find the container with id 75c58b355d0dcaba28c86365370b8d40391da1267f550ba8668fa6ee237367b6 Oct 30 22:01:01.150738 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:01:01.150707 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-6vzg4" Oct 30 22:01:01.151354 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:01:01.151330 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-6vzg4" Oct 30 22:01:01.155930 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:01:01.152316 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-6vzg4" Oct 30 22:01:01.794506 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:01:01.794467 5748 generic.go:356] "Generic (PLEG): container finished" podID="8e64a170-0a90-4e0b-b4ce-87e92d676ac1" containerID="6be82cad2f5e56dc0d2d5e022fa2b7b2de14ba4eabdf60c958b2aea6cd1aae8d" exitCode=0 Oct 30 22:01:01.794687 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:01:01.794543 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71a3d788-tb49x" event={"ID":"8e64a170-0a90-4e0b-b4ce-87e92d676ac1","Type":"ContainerDied","Data":"6be82cad2f5e56dc0d2d5e022fa2b7b2de14ba4eabdf60c958b2aea6cd1aae8d"} Oct 30 22:01:01.794687 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:01:01.794586 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71a3d788-tb49x" event={"ID":"8e64a170-0a90-4e0b-b4ce-87e92d676ac1","Type":"ContainerStarted","Data":"75c58b355d0dcaba28c86365370b8d40391da1267f550ba8668fa6ee237367b6"} Oct 30 22:01:02.961826 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:01:02.961803 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71a3d788-tb49x" Oct 30 22:01:02.993710 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:01:02.993680 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e64a170-0a90-4e0b-b4ce-87e92d676ac1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e64a170-0a90-4e0b-b4ce-87e92d676ac1-maas-api-openshift-ca\") pod \"8e64a170-0a90-4e0b-b4ce-87e92d676ac1\" (UID: \"8e64a170-0a90-4e0b-b4ce-87e92d676ac1\") " Oct 30 22:01:02.994076 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:01:02.993731 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e64a170-0a90-4e0b-b4ce-87e92d676ac1-kube-api-access-tw6kl\" (UniqueName: \"kubernetes.io/projected/8e64a170-0a90-4e0b-b4ce-87e92d676ac1-kube-api-access-tw6kl\") pod \"8e64a170-0a90-4e0b-b4ce-87e92d676ac1\" (UID: \"8e64a170-0a90-4e0b-b4ce-87e92d676ac1\") " Oct 30 22:01:02.994076 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:01:02.994007 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e64a170-0a90-4e0b-b4ce-87e92d676ac1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e64a170-0a90-4e0b-b4ce-87e92d676ac1-maas-api-xks-ca\") pod \"8e64a170-0a90-4e0b-b4ce-87e92d676ac1\" (UID: \"8e64a170-0a90-4e0b-b4ce-87e92d676ac1\") " Oct 30 22:01:02.994228 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:01:02.994072 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e64a170-0a90-4e0b-b4ce-87e92d676ac1-maas-api-openshift-ca" pod "8e64a170-0a90-4e0b-b4ce-87e92d676ac1" (UID: "8e64a170-0a90-4e0b-b4ce-87e92d676ac1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:01:02.994499 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:01:02.994473 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e64a170-0a90-4e0b-b4ce-87e92d676ac1-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:01:02.995910 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:01:02.995872 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e64a170-0a90-4e0b-b4ce-87e92d676ac1-kube-api-access-tw6kl" pod "8e64a170-0a90-4e0b-b4ce-87e92d676ac1" (UID: "8e64a170-0a90-4e0b-b4ce-87e92d676ac1"). InnerVolumeSpecName "kube-api-access-tw6kl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:01:02.996011 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:01:02.995960 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e64a170-0a90-4e0b-b4ce-87e92d676ac1-maas-api-xks-ca" pod "8e64a170-0a90-4e0b-b4ce-87e92d676ac1" (UID: "8e64a170-0a90-4e0b-b4ce-87e92d676ac1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:01:03.095491 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:01:03.095409 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e64a170-0a90-4e0b-b4ce-87e92d676ac1-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:01:03.095491 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:01:03.095438 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw6kl\" (UniqueName: \"kubernetes.io/projected/8e64a170-0a90-4e0b-b4ce-87e92d676ac1-kube-api-access-tw6kl\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:01:03.807682 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:01:03.807649 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71a3d788-tb49x" Oct 30 22:01:03.807863 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:01:03.807693 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71a3d788-tb49x" event={"ID":"8e64a170-0a90-4e0b-b4ce-87e92d676ac1","Type":"ContainerDied","Data":"75c58b355d0dcaba28c86365370b8d40391da1267f550ba8668fa6ee237367b6"} Oct 30 22:01:03.807863 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:01:03.807741 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75c58b355d0dcaba28c86365370b8d40391da1267f550ba8668fa6ee237367b6" Oct 30 22:01:30.146923 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:01:30.146538 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:01:37.889922 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:01:37.888690 5748 scope.go:122] "RemoveContainer" containerID="317085ad620292f235705cd0094cd09a56dbb16ec4f4225c6233ddfeb0389b41" Oct 30 22:01:42.100461 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:01:42.100423 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57ae4154-bjptg"] Oct 30 22:01:42.103913 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:01:42.103871 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57ae4154-bjptg"] Oct 30 22:01:42.416911 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:01:42.416870 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="608b9e09-4acb-46eb-9ff9-f99d2fe04ce4" path="/var/lib/kubelet/pods/608b9e09-4acb-46eb-9ff9-f99d2fe04ce4/volumes" Oct 30 22:01:53.053302 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:01:53.053264 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2cf27b7-mkbc8"] Oct 30 22:01:53.103634 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:01:53.103596 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2cf27b7-mkbc8"] Oct 30 22:01:54.417349 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:01:54.417315 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="946ae176-398f-4cdd-a2cd-fa71edfb6463" path="/var/lib/kubelet/pods/946ae176-398f-4cdd-a2cd-fa71edfb6463/volumes" Oct 30 22:02:00.161102 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:02:00.161062 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:02:02.331957 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:02.331923 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71a3d788-tb49x"] Oct 30 22:02:02.382381 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:02.382350 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71a3d788-tb49x"] Oct 30 22:02:02.418409 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:02.418371 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e64a170-0a90-4e0b-b4ce-87e92d676ac1" path="/var/lib/kubelet/pods/8e64a170-0a90-4e0b-b4ce-87e92d676ac1/volumes" Oct 30 22:02:09.263179 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:09.263142 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-91874e-p4jlh"] Oct 30 22:02:09.263681 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:09.263667 5748 cpu_manager.go:414] "containerMap: removing container" podUID="8e64a170-0a90-4e0b-b4ce-87e92d676ac1" containerName="revoke-keys" Oct 30 22:02:09.263732 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:09.263682 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e64a170-0a90-4e0b-b4ce-87e92d676ac1" containerName="revoke-keys" Oct 30 22:02:09.263825 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:09.263815 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e64a170-0a90-4e0b-b4ce-87e92d676ac1" containerName="revoke-keys" Oct 30 22:02:09.266995 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:09.266979 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-91874e-p4jlh" Oct 30 22:02:09.373292 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:09.373265 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:02:09.373460 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:09.373307 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:02:09.373460 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:09.373420 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 22:02:09.373588 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:09.373529 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:02:09.376117 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:09.376096 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-91874e-p4jlh"] Oct 30 22:02:09.400906 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:09.400878 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4scm9\" (UniqueName: \"kubernetes.io/projected/fbabbf0d-88f2-4775-ab15-44a80c63d31d-kube-api-access-4scm9\") pod \"maas-api-revoke-keys-e2e-worker-w2-91874e-p4jlh\" (UID: \"fbabbf0d-88f2-4775-ab15-44a80c63d31d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-91874e-p4jlh" Oct 30 22:02:09.401055 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:09.401034 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbabbf0d-88f2-4775-ab15-44a80c63d31d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-91874e-p4jlh\" (UID: \"fbabbf0d-88f2-4775-ab15-44a80c63d31d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-91874e-p4jlh" Oct 30 22:02:09.401103 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:09.401088 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbabbf0d-88f2-4775-ab15-44a80c63d31d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-91874e-p4jlh\" (UID: \"fbabbf0d-88f2-4775-ab15-44a80c63d31d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-91874e-p4jlh" Oct 30 22:02:09.502299 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:09.502274 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbabbf0d-88f2-4775-ab15-44a80c63d31d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-91874e-p4jlh\" (UID: \"fbabbf0d-88f2-4775-ab15-44a80c63d31d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-91874e-p4jlh" Oct 30 22:02:09.502467 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:09.502444 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbabbf0d-88f2-4775-ab15-44a80c63d31d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-91874e-p4jlh\" (UID: \"fbabbf0d-88f2-4775-ab15-44a80c63d31d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-91874e-p4jlh" Oct 30 22:02:09.502671 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:09.502642 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4scm9\" (UniqueName: \"kubernetes.io/projected/fbabbf0d-88f2-4775-ab15-44a80c63d31d-kube-api-access-4scm9\") pod \"maas-api-revoke-keys-e2e-worker-w2-91874e-p4jlh\" (UID: \"fbabbf0d-88f2-4775-ab15-44a80c63d31d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-91874e-p4jlh" Oct 30 22:02:09.503098 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:09.503074 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbabbf0d-88f2-4775-ab15-44a80c63d31d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-91874e-p4jlh\" (UID: \"fbabbf0d-88f2-4775-ab15-44a80c63d31d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-91874e-p4jlh" Oct 30 22:02:09.504984 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:09.504959 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbabbf0d-88f2-4775-ab15-44a80c63d31d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-91874e-p4jlh\" (UID: \"fbabbf0d-88f2-4775-ab15-44a80c63d31d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-91874e-p4jlh" Oct 30 22:02:09.565222 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:09.565165 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4scm9\" (UniqueName: \"kubernetes.io/projected/fbabbf0d-88f2-4775-ab15-44a80c63d31d-kube-api-access-4scm9\") pod \"maas-api-revoke-keys-e2e-worker-w2-91874e-p4jlh\" (UID: \"fbabbf0d-88f2-4775-ab15-44a80c63d31d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-91874e-p4jlh" Oct 30 22:02:09.580177 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:09.580153 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-91874e-p4jlh" Oct 30 22:02:09.959776 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:09.959748 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-91874e-p4jlh"] Oct 30 22:02:09.961657 ip-10-0-143-61 kubenswrapper[5748]: W1030 22:02:09.961630 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbabbf0d_88f2_4775_ab15_44a80c63d31d.slice/crio-ccd05031ed0ef5446d3d597019e7c2c5f39a757814e8b5fd7689c9472536a0b4 WatchSource:0}: Error finding container ccd05031ed0ef5446d3d597019e7c2c5f39a757814e8b5fd7689c9472536a0b4: Status 404 returned error can't find the container with id ccd05031ed0ef5446d3d597019e7c2c5f39a757814e8b5fd7689c9472536a0b4 Oct 30 22:02:10.357861 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:10.357830 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-91874e-p4jlh" event={"ID":"fbabbf0d-88f2-4775-ab15-44a80c63d31d","Type":"ContainerStarted","Data":"815a6e6beab44301d0edcb0d11cae079e9a2f80884c78d70187fb7eac9390da3"} Oct 30 22:02:10.358240 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:10.357884 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-91874e-p4jlh" event={"ID":"fbabbf0d-88f2-4775-ab15-44a80c63d31d","Type":"ContainerStarted","Data":"ccd05031ed0ef5446d3d597019e7c2c5f39a757814e8b5fd7689c9472536a0b4"} Oct 30 22:02:10.476471 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:10.476373 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-91874e-p4jlh" podStartSLOduration=1.476357714 podStartE2EDuration="1.476357714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:10.474664613 +0000 UTC m=+2314.217185549" watchObservedRunningTime="2026-10-30 22:02:10.476357714 +0000 UTC m=+2314.218878649" Oct 30 22:02:11.365855 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:11.365771 5748 generic.go:356] "Generic (PLEG): container finished" podID="fbabbf0d-88f2-4775-ab15-44a80c63d31d" containerID="815a6e6beab44301d0edcb0d11cae079e9a2f80884c78d70187fb7eac9390da3" exitCode=0 Oct 30 22:02:11.366241 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:11.365861 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-91874e-p4jlh" event={"ID":"fbabbf0d-88f2-4775-ab15-44a80c63d31d","Type":"ContainerDied","Data":"815a6e6beab44301d0edcb0d11cae079e9a2f80884c78d70187fb7eac9390da3"} Oct 30 22:02:12.539132 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:12.539103 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-91874e-p4jlh" Oct 30 22:02:12.647163 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:12.647129 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbabbf0d-88f2-4775-ab15-44a80c63d31d-kube-api-access-4scm9\" (UniqueName: \"kubernetes.io/projected/fbabbf0d-88f2-4775-ab15-44a80c63d31d-kube-api-access-4scm9\") pod \"fbabbf0d-88f2-4775-ab15-44a80c63d31d\" (UID: \"fbabbf0d-88f2-4775-ab15-44a80c63d31d\") " Oct 30 22:02:12.647355 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:12.647321 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbabbf0d-88f2-4775-ab15-44a80c63d31d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbabbf0d-88f2-4775-ab15-44a80c63d31d-maas-api-openshift-ca\") pod \"fbabbf0d-88f2-4775-ab15-44a80c63d31d\" (UID: \"fbabbf0d-88f2-4775-ab15-44a80c63d31d\") " Oct 30 22:02:12.647454 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:12.647439 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbabbf0d-88f2-4775-ab15-44a80c63d31d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbabbf0d-88f2-4775-ab15-44a80c63d31d-maas-api-xks-ca\") pod \"fbabbf0d-88f2-4775-ab15-44a80c63d31d\" (UID: \"fbabbf0d-88f2-4775-ab15-44a80c63d31d\") " Oct 30 22:02:12.647746 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:12.647717 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbabbf0d-88f2-4775-ab15-44a80c63d31d-maas-api-openshift-ca" pod "fbabbf0d-88f2-4775-ab15-44a80c63d31d" (UID: "fbabbf0d-88f2-4775-ab15-44a80c63d31d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:12.648066 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:12.648044 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbabbf0d-88f2-4775-ab15-44a80c63d31d-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:12.649362 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:12.649338 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbabbf0d-88f2-4775-ab15-44a80c63d31d-kube-api-access-4scm9" pod "fbabbf0d-88f2-4775-ab15-44a80c63d31d" (UID: "fbabbf0d-88f2-4775-ab15-44a80c63d31d"). InnerVolumeSpecName "kube-api-access-4scm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:12.649466 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:12.649389 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbabbf0d-88f2-4775-ab15-44a80c63d31d-maas-api-xks-ca" pod "fbabbf0d-88f2-4775-ab15-44a80c63d31d" (UID: "fbabbf0d-88f2-4775-ab15-44a80c63d31d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:02:12.749511 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:12.749480 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4scm9\" (UniqueName: \"kubernetes.io/projected/fbabbf0d-88f2-4775-ab15-44a80c63d31d-kube-api-access-4scm9\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:12.749511 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:12.749510 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbabbf0d-88f2-4775-ab15-44a80c63d31d-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:13.388859 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:13.388832 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-91874e-p4jlh" Oct 30 22:02:13.388859 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:13.388842 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-91874e-p4jlh" event={"ID":"fbabbf0d-88f2-4775-ab15-44a80c63d31d","Type":"ContainerDied","Data":"ccd05031ed0ef5446d3d597019e7c2c5f39a757814e8b5fd7689c9472536a0b4"} Oct 30 22:02:13.389092 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:13.388890 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ccd05031ed0ef5446d3d597019e7c2c5f39a757814e8b5fd7689c9472536a0b4" Oct 30 22:02:13.778075 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:13.777988 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-91874e/distinct-2-w2-91874e-kserve-69fcf64f44-qm96r"] Oct 30 22:02:13.783929 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:13.783886 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-91874e/distinct-w2-91874e-kserve-58d6bdfc75-bdp5z"] Oct 30 22:02:28.801137 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:28.801097 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-91874e/distinct-2-w2-91874e-kserve-69fcf64f44-qm96r" podUID="0cb16f33-b208-4c11-bb12-322829bd51e5" containerName="main" containerID="cri-o://d9b311dbf4041d0a7755d6b6dddfbdd65a0c4bcb499249ea28327477f14ea58e" gracePeriod=45 Oct 30 22:02:28.805208 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:28.805174 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-91874e/distinct-w2-91874e-kserve-58d6bdfc75-bdp5z" podUID="cb39b415-f406-47bd-b591-be7e09240f5a" containerName="main" containerID="cri-o://89d5456791ba0ef1f88716c7b0b4dffed0a13643555e94539d60efae8876768d" gracePeriod=45 Oct 30 22:02:29.083169 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:29.082807 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-91874e/distinct-2-w2-91874e-kserve-69fcf64f44-qm96r" Oct 30 22:02:29.086303 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:29.086284 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-91874e/distinct-w2-91874e-kserve-58d6bdfc75-bdp5z" Oct 30 22:02:29.160685 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:29.160662 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb39b415-f406-47bd-b591-be7e09240f5a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb39b415-f406-47bd-b591-be7e09240f5a-tmp-dir\") pod \"cb39b415-f406-47bd-b591-be7e09240f5a\" (UID: \"cb39b415-f406-47bd-b591-be7e09240f5a\") " Oct 30 22:02:29.160809 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:29.160702 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cb16f33-b208-4c11-bb12-322829bd51e5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cb16f33-b208-4c11-bb12-322829bd51e5-dshm\") pod \"0cb16f33-b208-4c11-bb12-322829bd51e5\" (UID: \"0cb16f33-b208-4c11-bb12-322829bd51e5\") " Oct 30 22:02:29.160809 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:29.160743 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb39b415-f406-47bd-b591-be7e09240f5a-home\" (UniqueName: \"kubernetes.io/empty-dir/cb39b415-f406-47bd-b591-be7e09240f5a-home\") pod \"cb39b415-f406-47bd-b591-be7e09240f5a\" (UID: \"cb39b415-f406-47bd-b591-be7e09240f5a\") " Oct 30 22:02:29.160809 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:29.160804 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cb16f33-b208-4c11-bb12-322829bd51e5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cb16f33-b208-4c11-bb12-322829bd51e5-model-cache\") pod \"0cb16f33-b208-4c11-bb12-322829bd51e5\" (UID: \"0cb16f33-b208-4c11-bb12-322829bd51e5\") " Oct 30 22:02:29.161003 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:29.160846 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cb16f33-b208-4c11-bb12-322829bd51e5-tls-certs\" (UniqueName: \"kubernetes.io/secret/0cb16f33-b208-4c11-bb12-322829bd51e5-tls-certs\") pod \"0cb16f33-b208-4c11-bb12-322829bd51e5\" (UID: \"0cb16f33-b208-4c11-bb12-322829bd51e5\") " Oct 30 22:02:29.161003 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:29.160885 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cb16f33-b208-4c11-bb12-322829bd51e5-kube-api-access-bsnt2\" (UniqueName: \"kubernetes.io/projected/0cb16f33-b208-4c11-bb12-322829bd51e5-kube-api-access-bsnt2\") pod \"0cb16f33-b208-4c11-bb12-322829bd51e5\" (UID: \"0cb16f33-b208-4c11-bb12-322829bd51e5\") " Oct 30 22:02:29.161109 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:29.161017 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cb16f33-b208-4c11-bb12-322829bd51e5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cb16f33-b208-4c11-bb12-322829bd51e5-tmp-dir\") pod \"0cb16f33-b208-4c11-bb12-322829bd51e5\" (UID: \"0cb16f33-b208-4c11-bb12-322829bd51e5\") " Oct 30 22:02:29.161109 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:29.160942 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb39b415-f406-47bd-b591-be7e09240f5a-tmp-dir" pod "cb39b415-f406-47bd-b591-be7e09240f5a" (UID: "cb39b415-f406-47bd-b591-be7e09240f5a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:29.161109 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:29.161032 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb39b415-f406-47bd-b591-be7e09240f5a-home" pod "cb39b415-f406-47bd-b591-be7e09240f5a" (UID: "cb39b415-f406-47bd-b591-be7e09240f5a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:29.161109 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:29.161059 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb39b415-f406-47bd-b591-be7e09240f5a-tls-certs\" (UniqueName: \"kubernetes.io/secret/cb39b415-f406-47bd-b591-be7e09240f5a-tls-certs\") pod \"cb39b415-f406-47bd-b591-be7e09240f5a\" (UID: \"cb39b415-f406-47bd-b591-be7e09240f5a\") " Oct 30 22:02:29.161322 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:29.161114 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cb16f33-b208-4c11-bb12-322829bd51e5-model-cache" pod "0cb16f33-b208-4c11-bb12-322829bd51e5" (UID: "0cb16f33-b208-4c11-bb12-322829bd51e5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:29.161322 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:29.161159 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb39b415-f406-47bd-b591-be7e09240f5a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb39b415-f406-47bd-b591-be7e09240f5a-model-cache\") pod \"cb39b415-f406-47bd-b591-be7e09240f5a\" (UID: \"cb39b415-f406-47bd-b591-be7e09240f5a\") " Oct 30 22:02:29.161322 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:29.161213 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb39b415-f406-47bd-b591-be7e09240f5a-kube-api-access-q5qp4\" (UniqueName: \"kubernetes.io/projected/cb39b415-f406-47bd-b591-be7e09240f5a-kube-api-access-q5qp4\") pod \"cb39b415-f406-47bd-b591-be7e09240f5a\" (UID: \"cb39b415-f406-47bd-b591-be7e09240f5a\") " Oct 30 22:02:29.161322 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:29.161261 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb39b415-f406-47bd-b591-be7e09240f5a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb39b415-f406-47bd-b591-be7e09240f5a-dshm\") pod \"cb39b415-f406-47bd-b591-be7e09240f5a\" (UID: \"cb39b415-f406-47bd-b591-be7e09240f5a\") " Oct 30 22:02:29.161540 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:29.161323 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cb16f33-b208-4c11-bb12-322829bd51e5-tmp-dir" pod "0cb16f33-b208-4c11-bb12-322829bd51e5" (UID: "0cb16f33-b208-4c11-bb12-322829bd51e5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:29.161540 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:29.161329 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cb16f33-b208-4c11-bb12-322829bd51e5-home\" (UniqueName: \"kubernetes.io/empty-dir/0cb16f33-b208-4c11-bb12-322829bd51e5-home\") pod \"0cb16f33-b208-4c11-bb12-322829bd51e5\" (UID: \"0cb16f33-b208-4c11-bb12-322829bd51e5\") " Oct 30 22:02:29.161540 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:29.161518 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cb16f33-b208-4c11-bb12-322829bd51e5-home" pod "0cb16f33-b208-4c11-bb12-322829bd51e5" (UID: "0cb16f33-b208-4c11-bb12-322829bd51e5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:29.161704 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:29.161637 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb39b415-f406-47bd-b591-be7e09240f5a-model-cache" pod "cb39b415-f406-47bd-b591-be7e09240f5a" (UID: "cb39b415-f406-47bd-b591-be7e09240f5a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:29.162011 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:29.161990 5748 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb39b415-f406-47bd-b591-be7e09240f5a-home\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:29.162100 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:29.162015 5748 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cb16f33-b208-4c11-bb12-322829bd51e5-model-cache\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:29.162100 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:29.162031 5748 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cb16f33-b208-4c11-bb12-322829bd51e5-tmp-dir\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:29.162100 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:29.162045 5748 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb39b415-f406-47bd-b591-be7e09240f5a-model-cache\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:29.162100 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:29.162057 5748 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cb16f33-b208-4c11-bb12-322829bd51e5-home\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:29.162100 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:29.162071 5748 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb39b415-f406-47bd-b591-be7e09240f5a-tmp-dir\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:29.163522 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:29.163483 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cb16f33-b208-4c11-bb12-322829bd51e5-dshm" pod "0cb16f33-b208-4c11-bb12-322829bd51e5" (UID: "0cb16f33-b208-4c11-bb12-322829bd51e5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:29.163640 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:29.163561 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb39b415-f406-47bd-b591-be7e09240f5a-tls-certs" pod "cb39b415-f406-47bd-b591-be7e09240f5a" (UID: "cb39b415-f406-47bd-b591-be7e09240f5a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:02:29.163709 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:29.163635 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cb16f33-b208-4c11-bb12-322829bd51e5-kube-api-access-bsnt2" pod "0cb16f33-b208-4c11-bb12-322829bd51e5" (UID: "0cb16f33-b208-4c11-bb12-322829bd51e5"). InnerVolumeSpecName "kube-api-access-bsnt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:29.163786 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:29.163767 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb39b415-f406-47bd-b591-be7e09240f5a-dshm" pod "cb39b415-f406-47bd-b591-be7e09240f5a" (UID: "cb39b415-f406-47bd-b591-be7e09240f5a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:29.164124 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:29.164103 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cb16f33-b208-4c11-bb12-322829bd51e5-tls-certs" pod "0cb16f33-b208-4c11-bb12-322829bd51e5" (UID: "0cb16f33-b208-4c11-bb12-322829bd51e5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:02:29.164224 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:29.164200 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb39b415-f406-47bd-b591-be7e09240f5a-kube-api-access-q5qp4" pod "cb39b415-f406-47bd-b591-be7e09240f5a" (UID: "cb39b415-f406-47bd-b591-be7e09240f5a"). InnerVolumeSpecName "kube-api-access-q5qp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:29.264166 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:29.264113 5748 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cb16f33-b208-4c11-bb12-322829bd51e5-dshm\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:29.264166 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:29.264151 5748 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cb16f33-b208-4c11-bb12-322829bd51e5-tls-certs\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:29.264166 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:29.264168 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsnt2\" (UniqueName: \"kubernetes.io/projected/0cb16f33-b208-4c11-bb12-322829bd51e5-kube-api-access-bsnt2\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:29.264442 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:29.264182 5748 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb39b415-f406-47bd-b591-be7e09240f5a-tls-certs\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:29.264442 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:29.264196 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5qp4\" (UniqueName: \"kubernetes.io/projected/cb39b415-f406-47bd-b591-be7e09240f5a-kube-api-access-q5qp4\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:29.264442 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:29.264208 5748 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb39b415-f406-47bd-b591-be7e09240f5a-dshm\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:29.498596 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:29.498561 5748 generic.go:356] "Generic (PLEG): container finished" podID="cb39b415-f406-47bd-b591-be7e09240f5a" containerID="89d5456791ba0ef1f88716c7b0b4dffed0a13643555e94539d60efae8876768d" exitCode=0 Oct 30 22:02:29.498772 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:29.498617 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-91874e/distinct-w2-91874e-kserve-58d6bdfc75-bdp5z" Oct 30 22:02:29.498772 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:29.498643 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-91874e/distinct-w2-91874e-kserve-58d6bdfc75-bdp5z" event={"ID":"cb39b415-f406-47bd-b591-be7e09240f5a","Type":"ContainerDied","Data":"89d5456791ba0ef1f88716c7b0b4dffed0a13643555e94539d60efae8876768d"} Oct 30 22:02:29.498772 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:29.498690 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-91874e/distinct-w2-91874e-kserve-58d6bdfc75-bdp5z" event={"ID":"cb39b415-f406-47bd-b591-be7e09240f5a","Type":"ContainerDied","Data":"8a5a495c5ca6a7db3187d75babf16f4fee50183997a159451e755faaf9d76ac6"} Oct 30 22:02:29.498772 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:29.498711 5748 scope.go:122] "RemoveContainer" containerID="89d5456791ba0ef1f88716c7b0b4dffed0a13643555e94539d60efae8876768d" Oct 30 22:02:29.505766 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:29.505736 5748 generic.go:356] "Generic (PLEG): container finished" podID="0cb16f33-b208-4c11-bb12-322829bd51e5" containerID="d9b311dbf4041d0a7755d6b6dddfbdd65a0c4bcb499249ea28327477f14ea58e" exitCode=0 Oct 30 22:02:29.505766 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:29.505772 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-91874e/distinct-2-w2-91874e-kserve-69fcf64f44-qm96r" Oct 30 22:02:29.506041 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:29.505792 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-91874e/distinct-2-w2-91874e-kserve-69fcf64f44-qm96r" event={"ID":"0cb16f33-b208-4c11-bb12-322829bd51e5","Type":"ContainerDied","Data":"d9b311dbf4041d0a7755d6b6dddfbdd65a0c4bcb499249ea28327477f14ea58e"} Oct 30 22:02:29.506041 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:29.505821 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-91874e/distinct-2-w2-91874e-kserve-69fcf64f44-qm96r" event={"ID":"0cb16f33-b208-4c11-bb12-322829bd51e5","Type":"ContainerDied","Data":"e3a19d7d1a0c64fc8b7481b10fbdd75284cf4a6b59dede991c44d470d0712eda"} Oct 30 22:02:29.516551 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:29.516536 5748 scope.go:122] "RemoveContainer" containerID="89d5456791ba0ef1f88716c7b0b4dffed0a13643555e94539d60efae8876768d" Oct 30 22:02:29.516801 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:02:29.516784 5748 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89d5456791ba0ef1f88716c7b0b4dffed0a13643555e94539d60efae8876768d\": container with ID starting with 89d5456791ba0ef1f88716c7b0b4dffed0a13643555e94539d60efae8876768d not found: ID does not exist" containerID="89d5456791ba0ef1f88716c7b0b4dffed0a13643555e94539d60efae8876768d" Oct 30 22:02:29.516887 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:29.516812 5748 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89d5456791ba0ef1f88716c7b0b4dffed0a13643555e94539d60efae8876768d"} err="failed to get container status \"89d5456791ba0ef1f88716c7b0b4dffed0a13643555e94539d60efae8876768d\": rpc error: code = NotFound desc = could not find container \"89d5456791ba0ef1f88716c7b0b4dffed0a13643555e94539d60efae8876768d\": container with ID starting with 89d5456791ba0ef1f88716c7b0b4dffed0a13643555e94539d60efae8876768d not found: ID does not exist" Oct 30 22:02:29.516887 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:29.516837 5748 scope.go:122] "RemoveContainer" containerID="d9b311dbf4041d0a7755d6b6dddfbdd65a0c4bcb499249ea28327477f14ea58e" Oct 30 22:02:29.526043 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:29.526027 5748 scope.go:122] "RemoveContainer" containerID="d9b311dbf4041d0a7755d6b6dddfbdd65a0c4bcb499249ea28327477f14ea58e" Oct 30 22:02:29.526265 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:02:29.526245 5748 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9b311dbf4041d0a7755d6b6dddfbdd65a0c4bcb499249ea28327477f14ea58e\": container with ID starting with d9b311dbf4041d0a7755d6b6dddfbdd65a0c4bcb499249ea28327477f14ea58e not found: ID does not exist" containerID="d9b311dbf4041d0a7755d6b6dddfbdd65a0c4bcb499249ea28327477f14ea58e" Oct 30 22:02:29.526309 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:29.526272 5748 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9b311dbf4041d0a7755d6b6dddfbdd65a0c4bcb499249ea28327477f14ea58e"} err="failed to get container status \"d9b311dbf4041d0a7755d6b6dddfbdd65a0c4bcb499249ea28327477f14ea58e\": rpc error: code = NotFound desc = could not find container \"d9b311dbf4041d0a7755d6b6dddfbdd65a0c4bcb499249ea28327477f14ea58e\": container with ID starting with d9b311dbf4041d0a7755d6b6dddfbdd65a0c4bcb499249ea28327477f14ea58e not found: ID does not exist" Oct 30 22:02:29.666045 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:29.666019 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-91874e/distinct-2-w2-91874e-kserve-69fcf64f44-qm96r"] Oct 30 22:02:29.671833 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:29.671811 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-91874e/distinct-2-w2-91874e-kserve-69fcf64f44-qm96r"] Oct 30 22:02:29.783187 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:29.783112 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-91874e/distinct-w2-91874e-kserve-58d6bdfc75-bdp5z"] Oct 30 22:02:29.787204 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:29.787179 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-91874e/distinct-w2-91874e-kserve-58d6bdfc75-bdp5z"] Oct 30 22:02:30.147915 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:02:30.147866 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:02:30.419380 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:30.419298 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cb16f33-b208-4c11-bb12-322829bd51e5" path="/var/lib/kubelet/pods/0cb16f33-b208-4c11-bb12-322829bd51e5/volumes" Oct 30 22:02:30.419927 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:30.419890 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb39b415-f406-47bd-b591-be7e09240f5a" path="/var/lib/kubelet/pods/cb39b415-f406-47bd-b591-be7e09240f5a/volumes" Oct 30 22:02:37.929057 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:37.929026 5748 scope.go:122] "RemoveContainer" containerID="28a4be276fa5fa913ef9879a8529e628a1c63e8fb303eb149b0eee4f2a323a00" Oct 30 22:02:37.938418 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:37.938396 5748 scope.go:122] "RemoveContainer" containerID="b2b02a331d3d71175637572a702adf1d3c98b39834b837a02248679b5f8e09fd" Oct 30 22:02:42.366332 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:42.366254 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f80439bf-drncz"] Oct 30 22:02:42.366843 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:42.366826 5748 cpu_manager.go:414] "containerMap: removing container" podUID="0cb16f33-b208-4c11-bb12-322829bd51e5" containerName="main" Oct 30 22:02:42.366893 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:42.366845 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cb16f33-b208-4c11-bb12-322829bd51e5" containerName="main" Oct 30 22:02:42.366893 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:42.366862 5748 cpu_manager.go:414] "containerMap: removing container" podUID="cb39b415-f406-47bd-b591-be7e09240f5a" containerName="main" Oct 30 22:02:42.366893 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:42.366867 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb39b415-f406-47bd-b591-be7e09240f5a" containerName="main" Oct 30 22:02:42.366893 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:42.366908 5748 cpu_manager.go:414] "containerMap: removing container" podUID="fbabbf0d-88f2-4775-ab15-44a80c63d31d" containerName="revoke-keys" Oct 30 22:02:42.366893 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:42.366917 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbabbf0d-88f2-4775-ab15-44a80c63d31d" containerName="revoke-keys" Oct 30 22:02:42.367094 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:42.367026 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbabbf0d-88f2-4775-ab15-44a80c63d31d" containerName="revoke-keys" Oct 30 22:02:42.367094 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:42.367040 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cb16f33-b208-4c11-bb12-322829bd51e5" containerName="main" Oct 30 22:02:42.367094 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:42.367048 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb39b415-f406-47bd-b591-be7e09240f5a" containerName="main" Oct 30 22:02:42.371760 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:42.371745 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f80439bf-drncz" Oct 30 22:02:42.423090 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:42.423062 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f487998b-98lsz"] Oct 30 22:02:42.427212 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:42.427190 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f487998b-98lsz" Oct 30 22:02:42.470808 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:42.470780 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:02:42.473515 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:42.473490 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f80439bf-drncz"] Oct 30 22:02:42.476764 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:42.476743 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:02:42.476860 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:42.476765 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:02:42.476860 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:42.476784 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 22:02:42.485592 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:42.485565 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f487998b-98lsz"] Oct 30 22:02:42.518957 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:42.518927 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a48970bd-1994-40cd-a7e1-e731848c7da1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f487998b-98lsz\" (UID: \"a48970bd-1994-40cd-a7e1-e731848c7da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f487998b-98lsz" Oct 30 22:02:42.519188 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:42.519164 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/353fc156-d3cc-4d53-bef6-581488884e82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f80439bf-drncz\" (UID: \"353fc156-d3cc-4d53-bef6-581488884e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f80439bf-drncz" Oct 30 22:02:42.519302 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:42.519240 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vttwz\" (UniqueName: \"kubernetes.io/projected/a48970bd-1994-40cd-a7e1-e731848c7da1-kube-api-access-vttwz\") pod \"maas-api-revoke-sub-f487998b-98lsz\" (UID: \"a48970bd-1994-40cd-a7e1-e731848c7da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f487998b-98lsz" Oct 30 22:02:42.519377 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:42.519312 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2wrq\" (UniqueName: \"kubernetes.io/projected/353fc156-d3cc-4d53-bef6-581488884e82-kube-api-access-g2wrq\") pod \"maas-api-revoke-sub-f80439bf-drncz\" (UID: \"353fc156-d3cc-4d53-bef6-581488884e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f80439bf-drncz" Oct 30 22:02:42.519377 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:42.519328 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/353fc156-d3cc-4d53-bef6-581488884e82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f80439bf-drncz\" (UID: \"353fc156-d3cc-4d53-bef6-581488884e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f80439bf-drncz" Oct 30 22:02:42.519492 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:42.519427 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a48970bd-1994-40cd-a7e1-e731848c7da1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f487998b-98lsz\" (UID: \"a48970bd-1994-40cd-a7e1-e731848c7da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f487998b-98lsz" Oct 30 22:02:42.620601 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:42.620537 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vttwz\" (UniqueName: \"kubernetes.io/projected/a48970bd-1994-40cd-a7e1-e731848c7da1-kube-api-access-vttwz\") pod \"maas-api-revoke-sub-f487998b-98lsz\" (UID: \"a48970bd-1994-40cd-a7e1-e731848c7da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f487998b-98lsz" Oct 30 22:02:42.620601 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:42.620586 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2wrq\" (UniqueName: \"kubernetes.io/projected/353fc156-d3cc-4d53-bef6-581488884e82-kube-api-access-g2wrq\") pod \"maas-api-revoke-sub-f80439bf-drncz\" (UID: \"353fc156-d3cc-4d53-bef6-581488884e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f80439bf-drncz" Oct 30 22:02:42.620817 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:42.620604 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/353fc156-d3cc-4d53-bef6-581488884e82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f80439bf-drncz\" (UID: \"353fc156-d3cc-4d53-bef6-581488884e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f80439bf-drncz" Oct 30 22:02:42.620817 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:42.620644 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a48970bd-1994-40cd-a7e1-e731848c7da1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f487998b-98lsz\" (UID: \"a48970bd-1994-40cd-a7e1-e731848c7da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f487998b-98lsz" Oct 30 22:02:42.620817 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:42.620770 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a48970bd-1994-40cd-a7e1-e731848c7da1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f487998b-98lsz\" (UID: \"a48970bd-1994-40cd-a7e1-e731848c7da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f487998b-98lsz" Oct 30 22:02:42.621024 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:42.620871 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/353fc156-d3cc-4d53-bef6-581488884e82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f80439bf-drncz\" (UID: \"353fc156-d3cc-4d53-bef6-581488884e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f80439bf-drncz" Oct 30 22:02:42.621439 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:42.621417 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/353fc156-d3cc-4d53-bef6-581488884e82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f80439bf-drncz\" (UID: \"353fc156-d3cc-4d53-bef6-581488884e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f80439bf-drncz" Oct 30 22:02:42.621546 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:42.621415 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a48970bd-1994-40cd-a7e1-e731848c7da1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f487998b-98lsz\" (UID: \"a48970bd-1994-40cd-a7e1-e731848c7da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f487998b-98lsz" Oct 30 22:02:42.623086 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:42.623068 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a48970bd-1994-40cd-a7e1-e731848c7da1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f487998b-98lsz\" (UID: \"a48970bd-1994-40cd-a7e1-e731848c7da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f487998b-98lsz" Oct 30 22:02:42.623153 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:42.623103 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/353fc156-d3cc-4d53-bef6-581488884e82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f80439bf-drncz\" (UID: \"353fc156-d3cc-4d53-bef6-581488884e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f80439bf-drncz" Oct 30 22:02:42.682252 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:42.682212 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2wrq\" (UniqueName: \"kubernetes.io/projected/353fc156-d3cc-4d53-bef6-581488884e82-kube-api-access-g2wrq\") pod \"maas-api-revoke-sub-f80439bf-drncz\" (UID: \"353fc156-d3cc-4d53-bef6-581488884e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f80439bf-drncz" Oct 30 22:02:42.682405 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:42.682261 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vttwz\" (UniqueName: \"kubernetes.io/projected/a48970bd-1994-40cd-a7e1-e731848c7da1-kube-api-access-vttwz\") pod \"maas-api-revoke-sub-f487998b-98lsz\" (UID: \"a48970bd-1994-40cd-a7e1-e731848c7da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f487998b-98lsz" Oct 30 22:02:42.685059 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:42.685041 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f80439bf-drncz" Oct 30 22:02:42.742557 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:42.742532 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f487998b-98lsz" Oct 30 22:02:43.062972 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:43.062942 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f80439bf-drncz"] Oct 30 22:02:43.064070 ip-10-0-143-61 kubenswrapper[5748]: W1030 22:02:43.064039 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod353fc156_d3cc_4d53_bef6_581488884e82.slice/crio-0bb5389ae9ab109bbe76d8c645c12f7ce03d65ae9743f7f4f40310a24e2d83d4 WatchSource:0}: Error finding container 0bb5389ae9ab109bbe76d8c645c12f7ce03d65ae9743f7f4f40310a24e2d83d4: Status 404 returned error can't find the container with id 0bb5389ae9ab109bbe76d8c645c12f7ce03d65ae9743f7f4f40310a24e2d83d4 Oct 30 22:02:43.165254 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:43.165224 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f487998b-98lsz"] Oct 30 22:02:43.166800 ip-10-0-143-61 kubenswrapper[5748]: W1030 22:02:43.166770 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda48970bd_1994_40cd_a7e1_e731848c7da1.slice/crio-5db926a75570dd00c4119ecf3d641cf321f6e285da5cae8f4fd88d9561d5a06f WatchSource:0}: Error finding container 5db926a75570dd00c4119ecf3d641cf321f6e285da5cae8f4fd88d9561d5a06f: Status 404 returned error can't find the container with id 5db926a75570dd00c4119ecf3d641cf321f6e285da5cae8f4fd88d9561d5a06f Oct 30 22:02:43.598403 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:43.598332 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f487998b-98lsz" event={"ID":"a48970bd-1994-40cd-a7e1-e731848c7da1","Type":"ContainerStarted","Data":"bedb2938b537b6b17a2f9aa5850f1342bae65dd4422070dedeb3c4290675bdbb"} Oct 30 22:02:43.598749 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:43.598400 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f487998b-98lsz" event={"ID":"a48970bd-1994-40cd-a7e1-e731848c7da1","Type":"ContainerStarted","Data":"5db926a75570dd00c4119ecf3d641cf321f6e285da5cae8f4fd88d9561d5a06f"} Oct 30 22:02:43.599966 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:43.599942 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f80439bf-drncz" event={"ID":"353fc156-d3cc-4d53-bef6-581488884e82","Type":"ContainerStarted","Data":"cc16939db61aef1fe585397b431c3c26de7582d0acd26707c77d41e6acda9007"} Oct 30 22:02:43.600087 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:43.599984 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f80439bf-drncz" event={"ID":"353fc156-d3cc-4d53-bef6-581488884e82","Type":"ContainerStarted","Data":"0bb5389ae9ab109bbe76d8c645c12f7ce03d65ae9743f7f4f40310a24e2d83d4"} Oct 30 22:02:43.718720 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:43.718668 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f487998b-98lsz" podStartSLOduration=1.718652772 podStartE2EDuration="1.718652772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:43.71573815 +0000 UTC m=+2347.458259078" watchObservedRunningTime="2026-10-30 22:02:43.718652772 +0000 UTC m=+2347.461173774" Oct 30 22:02:43.833688 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:43.833636 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f80439bf-drncz" podStartSLOduration=1.833622093 podStartE2EDuration="1.833622093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:43.832275728 +0000 UTC m=+2347.574796665" watchObservedRunningTime="2026-10-30 22:02:43.833622093 +0000 UTC m=+2347.576143071" Oct 30 22:02:44.607204 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:44.607170 5748 generic.go:356] "Generic (PLEG): container finished" podID="a48970bd-1994-40cd-a7e1-e731848c7da1" containerID="bedb2938b537b6b17a2f9aa5850f1342bae65dd4422070dedeb3c4290675bdbb" exitCode=0 Oct 30 22:02:44.607676 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:44.607254 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f487998b-98lsz" event={"ID":"a48970bd-1994-40cd-a7e1-e731848c7da1","Type":"ContainerDied","Data":"bedb2938b537b6b17a2f9aa5850f1342bae65dd4422070dedeb3c4290675bdbb"} Oct 30 22:02:44.608937 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:44.608914 5748 generic.go:356] "Generic (PLEG): container finished" podID="353fc156-d3cc-4d53-bef6-581488884e82" containerID="cc16939db61aef1fe585397b431c3c26de7582d0acd26707c77d41e6acda9007" exitCode=0 Oct 30 22:02:44.609075 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:44.608997 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f80439bf-drncz" event={"ID":"353fc156-d3cc-4d53-bef6-581488884e82","Type":"ContainerDied","Data":"cc16939db61aef1fe585397b431c3c26de7582d0acd26707c77d41e6acda9007"} Oct 30 22:02:45.936218 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:45.936194 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f80439bf-drncz" Oct 30 22:02:45.939887 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:45.939869 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f487998b-98lsz" Oct 30 22:02:46.071982 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:46.071953 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/353fc156-d3cc-4d53-bef6-581488884e82-kube-api-access-g2wrq\" (UniqueName: \"kubernetes.io/projected/353fc156-d3cc-4d53-bef6-581488884e82-kube-api-access-g2wrq\") pod \"353fc156-d3cc-4d53-bef6-581488884e82\" (UID: \"353fc156-d3cc-4d53-bef6-581488884e82\") " Oct 30 22:02:46.072143 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:46.072121 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a48970bd-1994-40cd-a7e1-e731848c7da1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a48970bd-1994-40cd-a7e1-e731848c7da1-maas-api-openshift-ca\") pod \"a48970bd-1994-40cd-a7e1-e731848c7da1\" (UID: \"a48970bd-1994-40cd-a7e1-e731848c7da1\") " Oct 30 22:02:46.072201 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:46.072152 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a48970bd-1994-40cd-a7e1-e731848c7da1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a48970bd-1994-40cd-a7e1-e731848c7da1-maas-api-xks-ca\") pod \"a48970bd-1994-40cd-a7e1-e731848c7da1\" (UID: \"a48970bd-1994-40cd-a7e1-e731848c7da1\") " Oct 30 22:02:46.072201 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:46.072174 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a48970bd-1994-40cd-a7e1-e731848c7da1-kube-api-access-vttwz\" (UniqueName: \"kubernetes.io/projected/a48970bd-1994-40cd-a7e1-e731848c7da1-kube-api-access-vttwz\") pod \"a48970bd-1994-40cd-a7e1-e731848c7da1\" (UID: \"a48970bd-1994-40cd-a7e1-e731848c7da1\") " Oct 30 22:02:46.072311 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:46.072250 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/353fc156-d3cc-4d53-bef6-581488884e82-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/353fc156-d3cc-4d53-bef6-581488884e82-maas-api-openshift-ca\") pod \"353fc156-d3cc-4d53-bef6-581488884e82\" (UID: \"353fc156-d3cc-4d53-bef6-581488884e82\") " Oct 30 22:02:46.072377 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:46.072311 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/353fc156-d3cc-4d53-bef6-581488884e82-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/353fc156-d3cc-4d53-bef6-581488884e82-maas-api-xks-ca\") pod \"353fc156-d3cc-4d53-bef6-581488884e82\" (UID: \"353fc156-d3cc-4d53-bef6-581488884e82\") " Oct 30 22:02:46.072691 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:46.072565 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a48970bd-1994-40cd-a7e1-e731848c7da1-maas-api-openshift-ca" pod "a48970bd-1994-40cd-a7e1-e731848c7da1" (UID: "a48970bd-1994-40cd-a7e1-e731848c7da1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:46.072849 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:46.072738 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/353fc156-d3cc-4d53-bef6-581488884e82-maas-api-openshift-ca" pod "353fc156-d3cc-4d53-bef6-581488884e82" (UID: "353fc156-d3cc-4d53-bef6-581488884e82"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:46.072967 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:46.072878 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a48970bd-1994-40cd-a7e1-e731848c7da1-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:46.072967 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:46.072922 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/353fc156-d3cc-4d53-bef6-581488884e82-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:46.074414 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:46.074377 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a48970bd-1994-40cd-a7e1-e731848c7da1-maas-api-xks-ca" pod "a48970bd-1994-40cd-a7e1-e731848c7da1" (UID: "a48970bd-1994-40cd-a7e1-e731848c7da1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:02:46.075436 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:46.075408 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a48970bd-1994-40cd-a7e1-e731848c7da1-kube-api-access-vttwz" pod "a48970bd-1994-40cd-a7e1-e731848c7da1" (UID: "a48970bd-1994-40cd-a7e1-e731848c7da1"). InnerVolumeSpecName "kube-api-access-vttwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:46.075600 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:46.075487 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/353fc156-d3cc-4d53-bef6-581488884e82-kube-api-access-g2wrq" pod "353fc156-d3cc-4d53-bef6-581488884e82" (UID: "353fc156-d3cc-4d53-bef6-581488884e82"). InnerVolumeSpecName "kube-api-access-g2wrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:46.075600 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:46.075526 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/353fc156-d3cc-4d53-bef6-581488884e82-maas-api-xks-ca" pod "353fc156-d3cc-4d53-bef6-581488884e82" (UID: "353fc156-d3cc-4d53-bef6-581488884e82"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:02:46.174229 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:46.174153 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/353fc156-d3cc-4d53-bef6-581488884e82-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:46.174229 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:46.174187 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2wrq\" (UniqueName: \"kubernetes.io/projected/353fc156-d3cc-4d53-bef6-581488884e82-kube-api-access-g2wrq\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:46.174229 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:46.174204 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a48970bd-1994-40cd-a7e1-e731848c7da1-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:46.174229 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:46.174219 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vttwz\" (UniqueName: \"kubernetes.io/projected/a48970bd-1994-40cd-a7e1-e731848c7da1-kube-api-access-vttwz\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:46.623594 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:46.623563 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f487998b-98lsz" Oct 30 22:02:46.623766 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:46.623600 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f487998b-98lsz" event={"ID":"a48970bd-1994-40cd-a7e1-e731848c7da1","Type":"ContainerDied","Data":"5db926a75570dd00c4119ecf3d641cf321f6e285da5cae8f4fd88d9561d5a06f"} Oct 30 22:02:46.623766 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:46.623655 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5db926a75570dd00c4119ecf3d641cf321f6e285da5cae8f4fd88d9561d5a06f" Oct 30 22:02:46.625027 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:46.625004 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f80439bf-drncz" event={"ID":"353fc156-d3cc-4d53-bef6-581488884e82","Type":"ContainerDied","Data":"0bb5389ae9ab109bbe76d8c645c12f7ce03d65ae9743f7f4f40310a24e2d83d4"} Oct 30 22:02:46.625158 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:46.625031 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f80439bf-drncz" Oct 30 22:02:46.625158 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:46.625040 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0bb5389ae9ab109bbe76d8c645c12f7ce03d65ae9743f7f4f40310a24e2d83d4" Oct 30 22:02:51.053820 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:51.053782 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v"] Oct 30 22:02:51.054388 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:51.054369 5748 cpu_manager.go:414] "containerMap: removing container" podUID="353fc156-d3cc-4d53-bef6-581488884e82" containerName="revoke-keys" Oct 30 22:02:51.054388 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:51.054383 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="353fc156-d3cc-4d53-bef6-581488884e82" containerName="revoke-keys" Oct 30 22:02:51.054528 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:51.054431 5748 cpu_manager.go:414] "containerMap: removing container" podUID="a48970bd-1994-40cd-a7e1-e731848c7da1" containerName="revoke-keys" Oct 30 22:02:51.054528 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:51.054441 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a48970bd-1994-40cd-a7e1-e731848c7da1" containerName="revoke-keys" Oct 30 22:02:51.054528 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:51.054526 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="353fc156-d3cc-4d53-bef6-581488884e82" containerName="revoke-keys" Oct 30 22:02:51.054689 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:51.054542 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="a48970bd-1994-40cd-a7e1-e731848c7da1" containerName="revoke-keys" Oct 30 22:02:51.058236 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:51.058214 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v" Oct 30 22:02:51.118937 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:51.118348 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-19bc92-openshift-default-dockercfg-pgp4f\"" Oct 30 22:02:51.170271 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:51.170236 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v"] Oct 30 22:02:51.249551 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:51.249510 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/024f007a-0f46-4093-8554-93a335039f10-istio-data\") pod \"e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v\" (UID: \"024f007a-0f46-4093-8554-93a335039f10\") " pod="openshift-ingress/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v" Oct 30 22:02:51.249750 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:51.249629 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/024f007a-0f46-4093-8554-93a335039f10-istio-podinfo\") pod \"e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v\" (UID: \"024f007a-0f46-4093-8554-93a335039f10\") " pod="openshift-ingress/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v" Oct 30 22:02:51.249819 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:51.249788 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/024f007a-0f46-4093-8554-93a335039f10-istiod-ca-cert\") pod \"e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v\" (UID: \"024f007a-0f46-4093-8554-93a335039f10\") " pod="openshift-ingress/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v" Oct 30 22:02:51.249923 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:51.249870 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/024f007a-0f46-4093-8554-93a335039f10-workload-socket\") pod \"e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v\" (UID: \"024f007a-0f46-4093-8554-93a335039f10\") " pod="openshift-ingress/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v" Oct 30 22:02:51.250026 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:51.250006 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/024f007a-0f46-4093-8554-93a335039f10-istio-token\") pod \"e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v\" (UID: \"024f007a-0f46-4093-8554-93a335039f10\") " pod="openshift-ingress/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v" Oct 30 22:02:51.250086 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:51.250063 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mvqz\" (UniqueName: \"kubernetes.io/projected/024f007a-0f46-4093-8554-93a335039f10-kube-api-access-9mvqz\") pod \"e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v\" (UID: \"024f007a-0f46-4093-8554-93a335039f10\") " pod="openshift-ingress/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v" Oct 30 22:02:51.250168 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:51.250153 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/024f007a-0f46-4093-8554-93a335039f10-workload-certs\") pod \"e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v\" (UID: \"024f007a-0f46-4093-8554-93a335039f10\") " pod="openshift-ingress/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v" Oct 30 22:02:51.250205 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:51.250183 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/024f007a-0f46-4093-8554-93a335039f10-istio-envoy\") pod \"e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v\" (UID: \"024f007a-0f46-4093-8554-93a335039f10\") " pod="openshift-ingress/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v" Oct 30 22:02:51.250238 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:51.250206 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/024f007a-0f46-4093-8554-93a335039f10-credential-socket\") pod \"e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v\" (UID: \"024f007a-0f46-4093-8554-93a335039f10\") " pod="openshift-ingress/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v" Oct 30 22:02:51.351403 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:51.351313 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/024f007a-0f46-4093-8554-93a335039f10-workload-socket\") pod \"e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v\" (UID: \"024f007a-0f46-4093-8554-93a335039f10\") " pod="openshift-ingress/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v" Oct 30 22:02:51.351403 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:51.351373 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/024f007a-0f46-4093-8554-93a335039f10-istio-token\") pod \"e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v\" (UID: \"024f007a-0f46-4093-8554-93a335039f10\") " pod="openshift-ingress/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v" Oct 30 22:02:51.351403 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:51.351402 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mvqz\" (UniqueName: \"kubernetes.io/projected/024f007a-0f46-4093-8554-93a335039f10-kube-api-access-9mvqz\") pod \"e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v\" (UID: \"024f007a-0f46-4093-8554-93a335039f10\") " pod="openshift-ingress/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v" Oct 30 22:02:51.351713 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:51.351432 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/024f007a-0f46-4093-8554-93a335039f10-workload-certs\") pod \"e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v\" (UID: \"024f007a-0f46-4093-8554-93a335039f10\") " pod="openshift-ingress/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v" Oct 30 22:02:51.351713 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:51.351453 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/024f007a-0f46-4093-8554-93a335039f10-istio-envoy\") pod \"e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v\" (UID: \"024f007a-0f46-4093-8554-93a335039f10\") " pod="openshift-ingress/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v" Oct 30 22:02:51.351713 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:51.351606 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/024f007a-0f46-4093-8554-93a335039f10-credential-socket\") pod \"e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v\" (UID: \"024f007a-0f46-4093-8554-93a335039f10\") " pod="openshift-ingress/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v" Oct 30 22:02:51.351713 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:51.351703 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/024f007a-0f46-4093-8554-93a335039f10-istio-data\") pod \"e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v\" (UID: \"024f007a-0f46-4093-8554-93a335039f10\") " pod="openshift-ingress/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v" Oct 30 22:02:51.352005 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:51.351768 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/024f007a-0f46-4093-8554-93a335039f10-istio-podinfo\") pod \"e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v\" (UID: \"024f007a-0f46-4093-8554-93a335039f10\") " pod="openshift-ingress/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v" Oct 30 22:02:51.352005 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:51.351865 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/024f007a-0f46-4093-8554-93a335039f10-workload-certs\") pod \"e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v\" (UID: \"024f007a-0f46-4093-8554-93a335039f10\") " pod="openshift-ingress/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v" Oct 30 22:02:51.352005 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:51.351887 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/024f007a-0f46-4093-8554-93a335039f10-istiod-ca-cert\") pod \"e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v\" (UID: \"024f007a-0f46-4093-8554-93a335039f10\") " pod="openshift-ingress/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v" Oct 30 22:02:51.352005 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:51.351958 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/024f007a-0f46-4093-8554-93a335039f10-credential-socket\") pod \"e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v\" (UID: \"024f007a-0f46-4093-8554-93a335039f10\") " pod="openshift-ingress/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v" Oct 30 22:02:51.352220 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:51.352134 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/024f007a-0f46-4093-8554-93a335039f10-workload-socket\") pod \"e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v\" (UID: \"024f007a-0f46-4093-8554-93a335039f10\") " pod="openshift-ingress/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v" Oct 30 22:02:51.352220 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:51.352170 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/024f007a-0f46-4093-8554-93a335039f10-istio-data\") pod \"e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v\" (UID: \"024f007a-0f46-4093-8554-93a335039f10\") " pod="openshift-ingress/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v" Oct 30 22:02:51.352595 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:51.352574 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/024f007a-0f46-4093-8554-93a335039f10-istiod-ca-cert\") pod \"e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v\" (UID: \"024f007a-0f46-4093-8554-93a335039f10\") " pod="openshift-ingress/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v" Oct 30 22:02:51.353744 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:51.353719 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/024f007a-0f46-4093-8554-93a335039f10-istio-envoy\") pod \"e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v\" (UID: \"024f007a-0f46-4093-8554-93a335039f10\") " pod="openshift-ingress/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v" Oct 30 22:02:51.354193 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:51.354176 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/024f007a-0f46-4093-8554-93a335039f10-istio-podinfo\") pod \"e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v\" (UID: \"024f007a-0f46-4093-8554-93a335039f10\") " pod="openshift-ingress/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v" Oct 30 22:02:51.419616 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:51.419587 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/024f007a-0f46-4093-8554-93a335039f10-istio-token\") pod \"e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v\" (UID: \"024f007a-0f46-4093-8554-93a335039f10\") " pod="openshift-ingress/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v" Oct 30 22:02:51.419830 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:51.419807 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mvqz\" (UniqueName: \"kubernetes.io/projected/024f007a-0f46-4093-8554-93a335039f10-kube-api-access-9mvqz\") pod \"e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v\" (UID: \"024f007a-0f46-4093-8554-93a335039f10\") " pod="openshift-ingress/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v" Oct 30 22:02:51.672438 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:51.672405 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v" Oct 30 22:02:52.119591 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:52.119562 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v"] Oct 30 22:02:52.121611 ip-10-0-143-61 kubenswrapper[5748]: W1030 22:02:52.121582 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod024f007a_0f46_4093_8554_93a335039f10.slice/crio-c38d058a1d57e95eca85b1add4fc61e2cb31e76766961d297d49e31bfdb66272 WatchSource:0}: Error finding container c38d058a1d57e95eca85b1add4fc61e2cb31e76766961d297d49e31bfdb66272: Status 404 returned error can't find the container with id c38d058a1d57e95eca85b1add4fc61e2cb31e76766961d297d49e31bfdb66272 Oct 30 22:02:52.124724 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:52.124684 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:02:52.124824 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:52.124778 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:02:52.124867 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:52.124822 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:02:52.665014 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:52.664978 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v" event={"ID":"024f007a-0f46-4093-8554-93a335039f10","Type":"ContainerStarted","Data":"6f596f846951d9760f05100004a2df05e925bb0ac478a90729c4d3a45d7c021a"} Oct 30 22:02:52.665205 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:52.665030 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v" event={"ID":"024f007a-0f46-4093-8554-93a335039f10","Type":"ContainerStarted","Data":"c38d058a1d57e95eca85b1add4fc61e2cb31e76766961d297d49e31bfdb66272"} Oct 30 22:02:52.673479 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:52.673456 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v" Oct 30 22:02:52.787731 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:52.787677 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v" podStartSLOduration=1.7876646059999999 podStartE2EDuration="1.787664606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:52.78618952 +0000 UTC m=+2356.528710455" watchObservedRunningTime="2026-10-30 22:02:52.787664606 +0000 UTC m=+2356.530185540" Oct 30 22:02:53.678655 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:53.678623 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v" Oct 30 22:02:54.678292 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:54.678211 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v" Oct 30 22:02:54.679279 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:02:54.679261 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v" Oct 30 22:03:00.144881 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:03:00.144841 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:03:04.494441 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:04.494395 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj"] Oct 30 22:03:04.499455 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:04.499429 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj" Oct 30 22:03:04.548593 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:04.548553 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-istio-envoy\") pod \"e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj\" (UID: \"4bb93eca-1d04-412d-aa83-2e1ac581bf1c\") " pod="openshift-ingress/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj" Oct 30 22:03:04.548786 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:04.548711 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-workload-socket\") pod \"e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj\" (UID: \"4bb93eca-1d04-412d-aa83-2e1ac581bf1c\") " pod="openshift-ingress/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj" Oct 30 22:03:04.548861 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:04.548785 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-credential-socket\") pod \"e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj\" (UID: \"4bb93eca-1d04-412d-aa83-2e1ac581bf1c\") " pod="openshift-ingress/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj" Oct 30 22:03:04.548861 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:04.548852 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-workload-certs\") pod \"e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj\" (UID: \"4bb93eca-1d04-412d-aa83-2e1ac581bf1c\") " pod="openshift-ingress/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj" Oct 30 22:03:04.548998 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:04.548882 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-istio-data\") pod \"e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj\" (UID: \"4bb93eca-1d04-412d-aa83-2e1ac581bf1c\") " pod="openshift-ingress/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj" Oct 30 22:03:04.548998 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:04.548934 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-istiod-ca-cert\") pod \"e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj\" (UID: \"4bb93eca-1d04-412d-aa83-2e1ac581bf1c\") " pod="openshift-ingress/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj" Oct 30 22:03:04.549110 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:04.549030 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-istio-token\") pod \"e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj\" (UID: \"4bb93eca-1d04-412d-aa83-2e1ac581bf1c\") " pod="openshift-ingress/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj" Oct 30 22:03:04.549171 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:04.549155 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-istio-podinfo\") pod \"e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj\" (UID: \"4bb93eca-1d04-412d-aa83-2e1ac581bf1c\") " pod="openshift-ingress/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj" Oct 30 22:03:04.549233 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:04.549204 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz9ct\" (UniqueName: \"kubernetes.io/projected/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-kube-api-access-rz9ct\") pod \"e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj\" (UID: \"4bb93eca-1d04-412d-aa83-2e1ac581bf1c\") " pod="openshift-ingress/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj" Oct 30 22:03:04.560382 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:04.560353 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj"] Oct 30 22:03:04.605788 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:04.605760 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-5bf4f8-openshift-default-dockercfg-zxq86\"" Oct 30 22:03:04.683705 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:04.683664 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-istio-envoy\") pod \"e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj\" (UID: \"4bb93eca-1d04-412d-aa83-2e1ac581bf1c\") " pod="openshift-ingress/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj" Oct 30 22:03:04.684049 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:04.684027 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-workload-socket\") pod \"e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj\" (UID: \"4bb93eca-1d04-412d-aa83-2e1ac581bf1c\") " pod="openshift-ingress/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj" Oct 30 22:03:04.684239 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:04.684220 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-credential-socket\") pod \"e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj\" (UID: \"4bb93eca-1d04-412d-aa83-2e1ac581bf1c\") " pod="openshift-ingress/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj" Oct 30 22:03:04.684412 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:04.684396 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-workload-certs\") pod \"e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj\" (UID: \"4bb93eca-1d04-412d-aa83-2e1ac581bf1c\") " pod="openshift-ingress/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj" Oct 30 22:03:04.684588 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:04.684570 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-istio-data\") pod \"e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj\" (UID: \"4bb93eca-1d04-412d-aa83-2e1ac581bf1c\") " pod="openshift-ingress/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj" Oct 30 22:03:04.684708 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:04.684693 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-istiod-ca-cert\") pod \"e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj\" (UID: \"4bb93eca-1d04-412d-aa83-2e1ac581bf1c\") " pod="openshift-ingress/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj" Oct 30 22:03:04.684910 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:04.684880 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-istio-token\") pod \"e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj\" (UID: \"4bb93eca-1d04-412d-aa83-2e1ac581bf1c\") " pod="openshift-ingress/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj" Oct 30 22:03:04.685138 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:04.685120 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-istio-podinfo\") pod \"e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj\" (UID: \"4bb93eca-1d04-412d-aa83-2e1ac581bf1c\") " pod="openshift-ingress/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj" Oct 30 22:03:04.685290 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:04.685273 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz9ct\" (UniqueName: \"kubernetes.io/projected/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-kube-api-access-rz9ct\") pod \"e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj\" (UID: \"4bb93eca-1d04-412d-aa83-2e1ac581bf1c\") " pod="openshift-ingress/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj" Oct 30 22:03:04.685767 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:04.685738 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-workload-certs\") pod \"e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj\" (UID: \"4bb93eca-1d04-412d-aa83-2e1ac581bf1c\") " pod="openshift-ingress/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj" Oct 30 22:03:04.694308 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:04.691964 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-istio-envoy\") pod \"e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj\" (UID: \"4bb93eca-1d04-412d-aa83-2e1ac581bf1c\") " pod="openshift-ingress/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj" Oct 30 22:03:04.694308 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:04.692233 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-workload-socket\") pod \"e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj\" (UID: \"4bb93eca-1d04-412d-aa83-2e1ac581bf1c\") " pod="openshift-ingress/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj" Oct 30 22:03:04.694308 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:04.692465 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-credential-socket\") pod \"e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj\" (UID: \"4bb93eca-1d04-412d-aa83-2e1ac581bf1c\") " pod="openshift-ingress/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj" Oct 30 22:03:04.694308 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:04.693207 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-istiod-ca-cert\") pod \"e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj\" (UID: \"4bb93eca-1d04-412d-aa83-2e1ac581bf1c\") " pod="openshift-ingress/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj" Oct 30 22:03:04.694308 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:04.693493 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-istio-data\") pod \"e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj\" (UID: \"4bb93eca-1d04-412d-aa83-2e1ac581bf1c\") " pod="openshift-ingress/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj" Oct 30 22:03:04.697204 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:04.697180 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-istio-podinfo\") pod \"e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj\" (UID: \"4bb93eca-1d04-412d-aa83-2e1ac581bf1c\") " pod="openshift-ingress/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj" Oct 30 22:03:04.752114 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:04.752045 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz9ct\" (UniqueName: \"kubernetes.io/projected/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-kube-api-access-rz9ct\") pod \"e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj\" (UID: \"4bb93eca-1d04-412d-aa83-2e1ac581bf1c\") " pod="openshift-ingress/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj" Oct 30 22:03:04.756212 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:04.756188 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-istio-token\") pod \"e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj\" (UID: \"4bb93eca-1d04-412d-aa83-2e1ac581bf1c\") " pod="openshift-ingress/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj" Oct 30 22:03:04.819435 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:04.818069 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj" Oct 30 22:03:05.260041 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:05.260016 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj"] Oct 30 22:03:05.261979 ip-10-0-143-61 kubenswrapper[5748]: W1030 22:03:05.261948 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bb93eca_1d04_412d_aa83_2e1ac581bf1c.slice/crio-3aab82d1fa99757cecf549d13cfbc42c0d561ad92ccd132753a82e2757d216e4 WatchSource:0}: Error finding container 3aab82d1fa99757cecf549d13cfbc42c0d561ad92ccd132753a82e2757d216e4: Status 404 returned error can't find the container with id 3aab82d1fa99757cecf549d13cfbc42c0d561ad92ccd132753a82e2757d216e4 Oct 30 22:03:05.264721 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:05.264684 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:03:05.264811 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:05.264754 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:03:05.264811 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:05.264788 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:03:05.753104 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:05.753067 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj" event={"ID":"4bb93eca-1d04-412d-aa83-2e1ac581bf1c","Type":"ContainerStarted","Data":"bb89cacb82ca662a89e11bea6558fa3b1b31b3ce407467306e5459cf2b88faea"} Oct 30 22:03:05.753589 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:05.753128 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj" event={"ID":"4bb93eca-1d04-412d-aa83-2e1ac581bf1c","Type":"ContainerStarted","Data":"3aab82d1fa99757cecf549d13cfbc42c0d561ad92ccd132753a82e2757d216e4"} Oct 30 22:03:05.819794 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:05.819759 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj" Oct 30 22:03:05.876437 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:05.876377 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj" podStartSLOduration=1.8763616060000001 podStartE2EDuration="1.876361606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:05.875885404 +0000 UTC m=+2369.618406352" watchObservedRunningTime="2026-10-30 22:03:05.876361606 +0000 UTC m=+2369.618882541" Oct 30 22:03:06.826284 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:06.826253 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj" Oct 30 22:03:07.770570 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:07.770534 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj" Oct 30 22:03:07.771475 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:07.771453 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj" Oct 30 22:03:23.047863 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:23.047826 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29ee56e3-xrcd6"] Oct 30 22:03:23.097926 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:23.097884 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29ee56e3-xrcd6"] Oct 30 22:03:24.417568 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:24.417529 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4814d07d-359c-4dd4-97de-c575b2b5052f" path="/var/lib/kubelet/pods/4814d07d-359c-4dd4-97de-c575b2b5052f/volumes" Oct 30 22:03:30.147046 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:03:30.147008 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:03:36.830594 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:36.830561 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j4v8r_3df036ff-1c00-4ce2-9e86-8acbeedf6c82/ovn-acl-logging/0.log" Oct 30 22:03:36.834794 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:36.834772 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j4v8r_3df036ff-1c00-4ce2-9e86-8acbeedf6c82/ovn-acl-logging/0.log" Oct 30 22:03:37.975049 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:37.975020 5748 scope.go:122] "RemoveContainer" containerID="19d37c912a14439fbe5c98c0c4a522bceed8a728aeaf170ea2f0784e236d1e31" Oct 30 22:03:37.984472 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:37.984444 5748 scope.go:122] "RemoveContainer" containerID="31c86636253bf4509a4063f7e5f34842af3902d12ab45bd08b64f61c4ee8c774" Oct 30 22:03:37.993032 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:37.993015 5748 scope.go:122] "RemoveContainer" containerID="2d3e4c676bc3a45af48d84132f8b4bcb45c7d54e18afd036a7a38f291aa92f7b" Oct 30 22:03:38.001544 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:38.001526 5748 scope.go:122] "RemoveContainer" containerID="31a5ff5b1532ac417134169487fe3ce80338a2a2dff7291ecd625f3e958b0c57" Oct 30 22:03:38.010694 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:38.010674 5748 scope.go:122] "RemoveContainer" containerID="48e28b1566427a8be6d0d320a258b03d26d24fa3a229b5e3edc97ffb9f1a5173" Oct 30 22:03:43.118619 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:43.118579 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-6vzg4"] Oct 30 22:03:43.119151 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:43.118852 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-6vzg4" podUID="0e9ffa82-546b-44a0-866b-44c89dfbb43e" containerName="istio-proxy" containerID="cri-o://1ddc6a01a5dc83bb1a2ff01f7c9f86bff070bc6be3f497c234534384fa0f0ab1" gracePeriod=30 Oct 30 22:03:43.151569 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:43.151532 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-cx2t5"] Oct 30 22:03:43.151874 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:43.151823 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-cx2t5" podUID="53df3a33-7883-4793-b495-e5c7543432f0" containerName="istio-proxy" containerID="cri-o://60220c4b1466575ccac22c1104f98508c572300d735da23cd540b5297a3a8211" gracePeriod=30 Oct 30 22:03:43.165526 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:43.165487 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-ps4v9"] Oct 30 22:03:43.165960 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:43.165912 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-ps4v9" podUID="1346c154-7a8b-42a7-96b1-cf9996e38008" containerName="istio-proxy" containerID="cri-o://8cb3695a13e4ebdeebf218c3794dcbeb97513ef215a55a9c1414662a6b29c1f7" gracePeriod=30 Oct 30 22:03:46.029120 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:46.029082 5748 patch_prober.go:29] interesting pod/e2e-mt-71a3d788-openshift-default-778558588f-ps4v9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:03:46.029581 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:46.029156 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-ps4v9" podUID="1346c154-7a8b-42a7-96b1-cf9996e38008" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:03:46.194120 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:46.194075 5748 patch_prober.go:29] interesting pod/e2e-mt-71a3d788-openshift-default-778558588f-6vzg4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:03:46.194351 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:46.194158 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-6vzg4" podUID="0e9ffa82-546b-44a0-866b-44c89dfbb43e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:03:46.337464 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:46.337386 5748 patch_prober.go:29] interesting pod/e2e-mt-71a3d788-openshift-default-778558588f-cx2t5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:03:46.337464 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:46.337439 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-cx2t5" podUID="53df3a33-7883-4793-b495-e5c7543432f0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:03:48.547870 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.547843 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-cx2t5" Oct 30 22:03:48.589695 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.589669 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-ps4v9" Oct 30 22:03:48.593041 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.593022 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-6vzg4" Oct 30 22:03:48.621738 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.621713 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e9ffa82-546b-44a0-866b-44c89dfbb43e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e9ffa82-546b-44a0-866b-44c89dfbb43e-istiod-ca-cert\") pod \"0e9ffa82-546b-44a0-866b-44c89dfbb43e\" (UID: \"0e9ffa82-546b-44a0-866b-44c89dfbb43e\") " Oct 30 22:03:48.621842 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.621751 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1346c154-7a8b-42a7-96b1-cf9996e38008-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1346c154-7a8b-42a7-96b1-cf9996e38008-istio-data\") pod \"1346c154-7a8b-42a7-96b1-cf9996e38008\" (UID: \"1346c154-7a8b-42a7-96b1-cf9996e38008\") " Oct 30 22:03:48.621842 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.621772 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e9ffa82-546b-44a0-866b-44c89dfbb43e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e9ffa82-546b-44a0-866b-44c89dfbb43e-istio-podinfo\") pod \"0e9ffa82-546b-44a0-866b-44c89dfbb43e\" (UID: \"0e9ffa82-546b-44a0-866b-44c89dfbb43e\") " Oct 30 22:03:48.621842 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.621800 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/53df3a33-7883-4793-b495-e5c7543432f0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53df3a33-7883-4793-b495-e5c7543432f0-istio-podinfo\") pod \"53df3a33-7883-4793-b495-e5c7543432f0\" (UID: \"53df3a33-7883-4793-b495-e5c7543432f0\") " Oct 30 22:03:48.622041 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.621851 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53df3a33-7883-4793-b495-e5c7543432f0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53df3a33-7883-4793-b495-e5c7543432f0-workload-socket\") pod \"53df3a33-7883-4793-b495-e5c7543432f0\" (UID: \"53df3a33-7883-4793-b495-e5c7543432f0\") " Oct 30 22:03:48.622041 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.621978 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e9ffa82-546b-44a0-866b-44c89dfbb43e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e9ffa82-546b-44a0-866b-44c89dfbb43e-istio-data\") pod \"0e9ffa82-546b-44a0-866b-44c89dfbb43e\" (UID: \"0e9ffa82-546b-44a0-866b-44c89dfbb43e\") " Oct 30 22:03:48.622041 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.621994 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1346c154-7a8b-42a7-96b1-cf9996e38008-istio-data" pod "1346c154-7a8b-42a7-96b1-cf9996e38008" (UID: "1346c154-7a8b-42a7-96b1-cf9996e38008"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:48.622041 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.622021 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e9ffa82-546b-44a0-866b-44c89dfbb43e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e9ffa82-546b-44a0-866b-44c89dfbb43e-istio-envoy\") pod \"0e9ffa82-546b-44a0-866b-44c89dfbb43e\" (UID: \"0e9ffa82-546b-44a0-866b-44c89dfbb43e\") " Oct 30 22:03:48.622258 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.622058 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1346c154-7a8b-42a7-96b1-cf9996e38008-kube-api-access-tgs6j\" (UniqueName: \"kubernetes.io/projected/1346c154-7a8b-42a7-96b1-cf9996e38008-kube-api-access-tgs6j\") pod \"1346c154-7a8b-42a7-96b1-cf9996e38008\" (UID: \"1346c154-7a8b-42a7-96b1-cf9996e38008\") " Oct 30 22:03:48.622258 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.622090 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53df3a33-7883-4793-b495-e5c7543432f0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53df3a33-7883-4793-b495-e5c7543432f0-istiod-ca-cert\") pod \"53df3a33-7883-4793-b495-e5c7543432f0\" (UID: \"53df3a33-7883-4793-b495-e5c7543432f0\") " Oct 30 22:03:48.622258 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.622116 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53df3a33-7883-4793-b495-e5c7543432f0-workload-socket" pod "53df3a33-7883-4793-b495-e5c7543432f0" (UID: "53df3a33-7883-4793-b495-e5c7543432f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:48.622258 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.622126 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53df3a33-7883-4793-b495-e5c7543432f0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53df3a33-7883-4793-b495-e5c7543432f0-istio-envoy\") pod \"53df3a33-7883-4793-b495-e5c7543432f0\" (UID: \"53df3a33-7883-4793-b495-e5c7543432f0\") " Oct 30 22:03:48.622258 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.622192 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53df3a33-7883-4793-b495-e5c7543432f0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53df3a33-7883-4793-b495-e5c7543432f0-credential-socket\") pod \"53df3a33-7883-4793-b495-e5c7543432f0\" (UID: \"53df3a33-7883-4793-b495-e5c7543432f0\") " Oct 30 22:03:48.622258 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.622241 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e9ffa82-546b-44a0-866b-44c89dfbb43e-istio-data" pod "0e9ffa82-546b-44a0-866b-44c89dfbb43e" (UID: "0e9ffa82-546b-44a0-866b-44c89dfbb43e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:48.622603 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.622245 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53df3a33-7883-4793-b495-e5c7543432f0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53df3a33-7883-4793-b495-e5c7543432f0-istio-data\") pod \"53df3a33-7883-4793-b495-e5c7543432f0\" (UID: \"53df3a33-7883-4793-b495-e5c7543432f0\") " Oct 30 22:03:48.622603 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.622295 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e9ffa82-546b-44a0-866b-44c89dfbb43e-istio-token\" (UniqueName: \"kubernetes.io/projected/0e9ffa82-546b-44a0-866b-44c89dfbb43e-istio-token\") pod \"0e9ffa82-546b-44a0-866b-44c89dfbb43e\" (UID: \"0e9ffa82-546b-44a0-866b-44c89dfbb43e\") " Oct 30 22:03:48.622603 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.622368 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1346c154-7a8b-42a7-96b1-cf9996e38008-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1346c154-7a8b-42a7-96b1-cf9996e38008-workload-certs\") pod \"1346c154-7a8b-42a7-96b1-cf9996e38008\" (UID: \"1346c154-7a8b-42a7-96b1-cf9996e38008\") " Oct 30 22:03:48.622603 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.622449 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1346c154-7a8b-42a7-96b1-cf9996e38008-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1346c154-7a8b-42a7-96b1-cf9996e38008-istiod-ca-cert\") pod \"1346c154-7a8b-42a7-96b1-cf9996e38008\" (UID: \"1346c154-7a8b-42a7-96b1-cf9996e38008\") " Oct 30 22:03:48.622603 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.622455 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53df3a33-7883-4793-b495-e5c7543432f0-istio-data" pod "53df3a33-7883-4793-b495-e5c7543432f0" (UID: "53df3a33-7883-4793-b495-e5c7543432f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:48.622603 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.622489 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e9ffa82-546b-44a0-866b-44c89dfbb43e-kube-api-access-pjvtd\" (UniqueName: \"kubernetes.io/projected/0e9ffa82-546b-44a0-866b-44c89dfbb43e-kube-api-access-pjvtd\") pod \"0e9ffa82-546b-44a0-866b-44c89dfbb43e\" (UID: \"0e9ffa82-546b-44a0-866b-44c89dfbb43e\") " Oct 30 22:03:48.622603 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.622527 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e9ffa82-546b-44a0-866b-44c89dfbb43e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e9ffa82-546b-44a0-866b-44c89dfbb43e-workload-certs\") pod \"0e9ffa82-546b-44a0-866b-44c89dfbb43e\" (UID: \"0e9ffa82-546b-44a0-866b-44c89dfbb43e\") " Oct 30 22:03:48.622603 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.622167 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e9ffa82-546b-44a0-866b-44c89dfbb43e-istiod-ca-cert" pod "0e9ffa82-546b-44a0-866b-44c89dfbb43e" (UID: "0e9ffa82-546b-44a0-866b-44c89dfbb43e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:48.622603 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.622562 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53df3a33-7883-4793-b495-e5c7543432f0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53df3a33-7883-4793-b495-e5c7543432f0-workload-certs\") pod \"53df3a33-7883-4793-b495-e5c7543432f0\" (UID: \"53df3a33-7883-4793-b495-e5c7543432f0\") " Oct 30 22:03:48.623123 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.622615 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1346c154-7a8b-42a7-96b1-cf9996e38008-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1346c154-7a8b-42a7-96b1-cf9996e38008-istio-podinfo\") pod \"1346c154-7a8b-42a7-96b1-cf9996e38008\" (UID: \"1346c154-7a8b-42a7-96b1-cf9996e38008\") " Oct 30 22:03:48.623123 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.622638 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53df3a33-7883-4793-b495-e5c7543432f0-credential-socket" pod "53df3a33-7883-4793-b495-e5c7543432f0" (UID: "53df3a33-7883-4793-b495-e5c7543432f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:48.623123 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.622669 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1346c154-7a8b-42a7-96b1-cf9996e38008-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1346c154-7a8b-42a7-96b1-cf9996e38008-credential-socket\") pod \"1346c154-7a8b-42a7-96b1-cf9996e38008\" (UID: \"1346c154-7a8b-42a7-96b1-cf9996e38008\") " Oct 30 22:03:48.623123 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.622721 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53df3a33-7883-4793-b495-e5c7543432f0-kube-api-access-gt7g7\" (UniqueName: \"kubernetes.io/projected/53df3a33-7883-4793-b495-e5c7543432f0-kube-api-access-gt7g7\") pod \"53df3a33-7883-4793-b495-e5c7543432f0\" (UID: \"53df3a33-7883-4793-b495-e5c7543432f0\") " Oct 30 22:03:48.623123 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.622734 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1346c154-7a8b-42a7-96b1-cf9996e38008-workload-certs" pod "1346c154-7a8b-42a7-96b1-cf9996e38008" (UID: "1346c154-7a8b-42a7-96b1-cf9996e38008"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:48.623123 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.622757 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1346c154-7a8b-42a7-96b1-cf9996e38008-istio-token\" (UniqueName: \"kubernetes.io/projected/1346c154-7a8b-42a7-96b1-cf9996e38008-istio-token\") pod \"1346c154-7a8b-42a7-96b1-cf9996e38008\" (UID: \"1346c154-7a8b-42a7-96b1-cf9996e38008\") " Oct 30 22:03:48.623123 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.622827 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1346c154-7a8b-42a7-96b1-cf9996e38008-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1346c154-7a8b-42a7-96b1-cf9996e38008-istio-envoy\") pod \"1346c154-7a8b-42a7-96b1-cf9996e38008\" (UID: \"1346c154-7a8b-42a7-96b1-cf9996e38008\") " Oct 30 22:03:48.623123 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.622863 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e9ffa82-546b-44a0-866b-44c89dfbb43e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e9ffa82-546b-44a0-866b-44c89dfbb43e-credential-socket\") pod \"0e9ffa82-546b-44a0-866b-44c89dfbb43e\" (UID: \"0e9ffa82-546b-44a0-866b-44c89dfbb43e\") " Oct 30 22:03:48.623123 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.622937 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e9ffa82-546b-44a0-866b-44c89dfbb43e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e9ffa82-546b-44a0-866b-44c89dfbb43e-workload-socket\") pod \"0e9ffa82-546b-44a0-866b-44c89dfbb43e\" (UID: \"0e9ffa82-546b-44a0-866b-44c89dfbb43e\") " Oct 30 22:03:48.623123 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.622961 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53df3a33-7883-4793-b495-e5c7543432f0-istiod-ca-cert" pod "53df3a33-7883-4793-b495-e5c7543432f0" (UID: "53df3a33-7883-4793-b495-e5c7543432f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:48.623123 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.622972 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1346c154-7a8b-42a7-96b1-cf9996e38008-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1346c154-7a8b-42a7-96b1-cf9996e38008-workload-socket\") pod \"1346c154-7a8b-42a7-96b1-cf9996e38008\" (UID: \"1346c154-7a8b-42a7-96b1-cf9996e38008\") " Oct 30 22:03:48.623123 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.623033 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53df3a33-7883-4793-b495-e5c7543432f0-istio-token\" (UniqueName: \"kubernetes.io/projected/53df3a33-7883-4793-b495-e5c7543432f0-istio-token\") pod \"53df3a33-7883-4793-b495-e5c7543432f0\" (UID: \"53df3a33-7883-4793-b495-e5c7543432f0\") " Oct 30 22:03:48.623760 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.623211 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1346c154-7a8b-42a7-96b1-cf9996e38008-credential-socket" pod "1346c154-7a8b-42a7-96b1-cf9996e38008" (UID: "1346c154-7a8b-42a7-96b1-cf9996e38008"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:48.623760 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.623714 5748 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1346c154-7a8b-42a7-96b1-cf9996e38008-credential-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:48.623760 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.623736 5748 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e9ffa82-546b-44a0-866b-44c89dfbb43e-istiod-ca-cert\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:48.623760 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.623751 5748 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1346c154-7a8b-42a7-96b1-cf9996e38008-istio-data\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:48.623994 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.623764 5748 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53df3a33-7883-4793-b495-e5c7543432f0-workload-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:48.623994 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.623778 5748 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e9ffa82-546b-44a0-866b-44c89dfbb43e-istio-data\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:48.623994 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.623793 5748 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53df3a33-7883-4793-b495-e5c7543432f0-istiod-ca-cert\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:48.623994 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.623807 5748 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53df3a33-7883-4793-b495-e5c7543432f0-credential-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:48.623994 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.623821 5748 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53df3a33-7883-4793-b495-e5c7543432f0-istio-data\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:48.623994 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.623834 5748 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1346c154-7a8b-42a7-96b1-cf9996e38008-workload-certs\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:48.624363 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.624193 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1346c154-7a8b-42a7-96b1-cf9996e38008-istiod-ca-cert" pod "1346c154-7a8b-42a7-96b1-cf9996e38008" (UID: "1346c154-7a8b-42a7-96b1-cf9996e38008"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:48.624940 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.624729 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1346c154-7a8b-42a7-96b1-cf9996e38008-workload-socket" pod "1346c154-7a8b-42a7-96b1-cf9996e38008" (UID: "1346c154-7a8b-42a7-96b1-cf9996e38008"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:48.624940 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.624847 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e9ffa82-546b-44a0-866b-44c89dfbb43e-credential-socket" pod "0e9ffa82-546b-44a0-866b-44c89dfbb43e" (UID: "0e9ffa82-546b-44a0-866b-44c89dfbb43e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:48.624940 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.624693 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/53df3a33-7883-4793-b495-e5c7543432f0-istio-podinfo" pod "53df3a33-7883-4793-b495-e5c7543432f0" (UID: "53df3a33-7883-4793-b495-e5c7543432f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:03:48.624940 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.624873 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e9ffa82-546b-44a0-866b-44c89dfbb43e-istio-podinfo" pod "0e9ffa82-546b-44a0-866b-44c89dfbb43e" (UID: "0e9ffa82-546b-44a0-866b-44c89dfbb43e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:03:48.625463 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.624953 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53df3a33-7883-4793-b495-e5c7543432f0-workload-certs" pod "53df3a33-7883-4793-b495-e5c7543432f0" (UID: "53df3a33-7883-4793-b495-e5c7543432f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:48.625463 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.624996 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e9ffa82-546b-44a0-866b-44c89dfbb43e-workload-socket" pod "0e9ffa82-546b-44a0-866b-44c89dfbb43e" (UID: "0e9ffa82-546b-44a0-866b-44c89dfbb43e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:48.625463 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.625237 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1346c154-7a8b-42a7-96b1-cf9996e38008-kube-api-access-tgs6j" pod "1346c154-7a8b-42a7-96b1-cf9996e38008" (UID: "1346c154-7a8b-42a7-96b1-cf9996e38008"). InnerVolumeSpecName "kube-api-access-tgs6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:48.625463 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.625327 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e9ffa82-546b-44a0-866b-44c89dfbb43e-workload-certs" pod "0e9ffa82-546b-44a0-866b-44c89dfbb43e" (UID: "0e9ffa82-546b-44a0-866b-44c89dfbb43e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:48.625684 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.625477 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e9ffa82-546b-44a0-866b-44c89dfbb43e-istio-envoy" pod "0e9ffa82-546b-44a0-866b-44c89dfbb43e" (UID: "0e9ffa82-546b-44a0-866b-44c89dfbb43e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:48.625684 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.625624 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53df3a33-7883-4793-b495-e5c7543432f0-istio-envoy" pod "53df3a33-7883-4793-b495-e5c7543432f0" (UID: "53df3a33-7883-4793-b495-e5c7543432f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:48.625800 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.625762 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53df3a33-7883-4793-b495-e5c7543432f0-istio-token" pod "53df3a33-7883-4793-b495-e5c7543432f0" (UID: "53df3a33-7883-4793-b495-e5c7543432f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:48.626516 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.626494 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e9ffa82-546b-44a0-866b-44c89dfbb43e-kube-api-access-pjvtd" pod "0e9ffa82-546b-44a0-866b-44c89dfbb43e" (UID: "0e9ffa82-546b-44a0-866b-44c89dfbb43e"). InnerVolumeSpecName "kube-api-access-pjvtd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:48.626668 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.626644 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1346c154-7a8b-42a7-96b1-cf9996e38008-istio-envoy" pod "1346c154-7a8b-42a7-96b1-cf9996e38008" (UID: "1346c154-7a8b-42a7-96b1-cf9996e38008"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:48.626747 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.626729 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e9ffa82-546b-44a0-866b-44c89dfbb43e-istio-token" pod "0e9ffa82-546b-44a0-866b-44c89dfbb43e" (UID: "0e9ffa82-546b-44a0-866b-44c89dfbb43e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:48.626991 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.626975 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1346c154-7a8b-42a7-96b1-cf9996e38008-istio-podinfo" pod "1346c154-7a8b-42a7-96b1-cf9996e38008" (UID: "1346c154-7a8b-42a7-96b1-cf9996e38008"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:03:48.628072 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.628048 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53df3a33-7883-4793-b495-e5c7543432f0-kube-api-access-gt7g7" pod "53df3a33-7883-4793-b495-e5c7543432f0" (UID: "53df3a33-7883-4793-b495-e5c7543432f0"). InnerVolumeSpecName "kube-api-access-gt7g7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:48.628163 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.628102 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1346c154-7a8b-42a7-96b1-cf9996e38008-istio-token" pod "1346c154-7a8b-42a7-96b1-cf9996e38008" (UID: "1346c154-7a8b-42a7-96b1-cf9996e38008"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:48.725155 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.725128 5748 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1346c154-7a8b-42a7-96b1-cf9996e38008-istiod-ca-cert\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:48.725312 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.725159 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjvtd\" (UniqueName: \"kubernetes.io/projected/0e9ffa82-546b-44a0-866b-44c89dfbb43e-kube-api-access-pjvtd\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:48.725312 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.725174 5748 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e9ffa82-546b-44a0-866b-44c89dfbb43e-workload-certs\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:48.725312 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.725189 5748 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53df3a33-7883-4793-b495-e5c7543432f0-workload-certs\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:48.725312 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.725202 5748 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1346c154-7a8b-42a7-96b1-cf9996e38008-istio-podinfo\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:48.725312 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.725214 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt7g7\" (UniqueName: \"kubernetes.io/projected/53df3a33-7883-4793-b495-e5c7543432f0-kube-api-access-gt7g7\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:48.725312 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.725228 5748 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1346c154-7a8b-42a7-96b1-cf9996e38008-istio-token\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:48.725312 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.725242 5748 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1346c154-7a8b-42a7-96b1-cf9996e38008-istio-envoy\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:48.725312 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.725255 5748 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e9ffa82-546b-44a0-866b-44c89dfbb43e-credential-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:48.725312 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.725269 5748 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e9ffa82-546b-44a0-866b-44c89dfbb43e-workload-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:48.725312 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.725283 5748 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1346c154-7a8b-42a7-96b1-cf9996e38008-workload-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:48.725312 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.725295 5748 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53df3a33-7883-4793-b495-e5c7543432f0-istio-token\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:48.725312 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.725309 5748 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e9ffa82-546b-44a0-866b-44c89dfbb43e-istio-podinfo\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:48.725866 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.725321 5748 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53df3a33-7883-4793-b495-e5c7543432f0-istio-podinfo\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:48.725866 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.725334 5748 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e9ffa82-546b-44a0-866b-44c89dfbb43e-istio-envoy\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:48.725866 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.725348 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgs6j\" (UniqueName: \"kubernetes.io/projected/1346c154-7a8b-42a7-96b1-cf9996e38008-kube-api-access-tgs6j\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:48.725866 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.725360 5748 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53df3a33-7883-4793-b495-e5c7543432f0-istio-envoy\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:48.725866 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:48.725373 5748 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e9ffa82-546b-44a0-866b-44c89dfbb43e-istio-token\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:49.092765 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:49.092678 5748 generic.go:356] "Generic (PLEG): container finished" podID="53df3a33-7883-4793-b495-e5c7543432f0" containerID="60220c4b1466575ccac22c1104f98508c572300d735da23cd540b5297a3a8211" exitCode=0 Oct 30 22:03:49.092765 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:49.092728 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-cx2t5" Oct 30 22:03:49.093013 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:49.092766 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-cx2t5" event={"ID":"53df3a33-7883-4793-b495-e5c7543432f0","Type":"ContainerDied","Data":"60220c4b1466575ccac22c1104f98508c572300d735da23cd540b5297a3a8211"} Oct 30 22:03:49.093013 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:49.092823 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-cx2t5" event={"ID":"53df3a33-7883-4793-b495-e5c7543432f0","Type":"ContainerDied","Data":"eb5cd5e03a9aa92e1e171ea7e4105632e2cb30de312f82b7c84e216e0cb4c4e1"} Oct 30 22:03:49.093013 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:49.092855 5748 scope.go:122] "RemoveContainer" containerID="60220c4b1466575ccac22c1104f98508c572300d735da23cd540b5297a3a8211" Oct 30 22:03:49.094250 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:49.094223 5748 generic.go:356] "Generic (PLEG): container finished" podID="1346c154-7a8b-42a7-96b1-cf9996e38008" containerID="8cb3695a13e4ebdeebf218c3794dcbeb97513ef215a55a9c1414662a6b29c1f7" exitCode=0 Oct 30 22:03:49.094363 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:49.094280 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-ps4v9" Oct 30 22:03:49.094363 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:49.094302 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-ps4v9" event={"ID":"1346c154-7a8b-42a7-96b1-cf9996e38008","Type":"ContainerDied","Data":"8cb3695a13e4ebdeebf218c3794dcbeb97513ef215a55a9c1414662a6b29c1f7"} Oct 30 22:03:49.094483 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:49.094367 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-ps4v9" event={"ID":"1346c154-7a8b-42a7-96b1-cf9996e38008","Type":"ContainerDied","Data":"7f6adb530241e3d292b6370d8374f55b54945581bdf3be02ac7a175747f21a99"} Oct 30 22:03:49.096172 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:49.096145 5748 generic.go:356] "Generic (PLEG): container finished" podID="0e9ffa82-546b-44a0-866b-44c89dfbb43e" containerID="1ddc6a01a5dc83bb1a2ff01f7c9f86bff070bc6be3f497c234534384fa0f0ab1" exitCode=0 Oct 30 22:03:49.096322 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:49.096304 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-6vzg4" event={"ID":"0e9ffa82-546b-44a0-866b-44c89dfbb43e","Type":"ContainerDied","Data":"1ddc6a01a5dc83bb1a2ff01f7c9f86bff070bc6be3f497c234534384fa0f0ab1"} Oct 30 22:03:49.096401 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:49.096347 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-6vzg4" event={"ID":"0e9ffa82-546b-44a0-866b-44c89dfbb43e","Type":"ContainerDied","Data":"5e14cadecb4d189ef4d14834a35eeed77aba063ec14af4ab555448243259daad"} Oct 30 22:03:49.096457 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:49.096413 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-6vzg4" Oct 30 22:03:49.104327 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:49.104307 5748 scope.go:122] "RemoveContainer" containerID="60220c4b1466575ccac22c1104f98508c572300d735da23cd540b5297a3a8211" Oct 30 22:03:49.104598 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:03:49.104580 5748 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60220c4b1466575ccac22c1104f98508c572300d735da23cd540b5297a3a8211\": container with ID starting with 60220c4b1466575ccac22c1104f98508c572300d735da23cd540b5297a3a8211 not found: ID does not exist" containerID="60220c4b1466575ccac22c1104f98508c572300d735da23cd540b5297a3a8211" Oct 30 22:03:49.104656 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:49.104605 5748 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60220c4b1466575ccac22c1104f98508c572300d735da23cd540b5297a3a8211"} err="failed to get container status \"60220c4b1466575ccac22c1104f98508c572300d735da23cd540b5297a3a8211\": rpc error: code = NotFound desc = could not find container \"60220c4b1466575ccac22c1104f98508c572300d735da23cd540b5297a3a8211\": container with ID starting with 60220c4b1466575ccac22c1104f98508c572300d735da23cd540b5297a3a8211 not found: ID does not exist" Oct 30 22:03:49.104656 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:49.104621 5748 scope.go:122] "RemoveContainer" containerID="8cb3695a13e4ebdeebf218c3794dcbeb97513ef215a55a9c1414662a6b29c1f7" Oct 30 22:03:49.115992 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:49.115973 5748 scope.go:122] "RemoveContainer" containerID="8cb3695a13e4ebdeebf218c3794dcbeb97513ef215a55a9c1414662a6b29c1f7" Oct 30 22:03:49.116286 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:03:49.116261 5748 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cb3695a13e4ebdeebf218c3794dcbeb97513ef215a55a9c1414662a6b29c1f7\": container with ID starting with 8cb3695a13e4ebdeebf218c3794dcbeb97513ef215a55a9c1414662a6b29c1f7 not found: ID does not exist" containerID="8cb3695a13e4ebdeebf218c3794dcbeb97513ef215a55a9c1414662a6b29c1f7" Oct 30 22:03:49.116350 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:49.116298 5748 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cb3695a13e4ebdeebf218c3794dcbeb97513ef215a55a9c1414662a6b29c1f7"} err="failed to get container status \"8cb3695a13e4ebdeebf218c3794dcbeb97513ef215a55a9c1414662a6b29c1f7\": rpc error: code = NotFound desc = could not find container \"8cb3695a13e4ebdeebf218c3794dcbeb97513ef215a55a9c1414662a6b29c1f7\": container with ID starting with 8cb3695a13e4ebdeebf218c3794dcbeb97513ef215a55a9c1414662a6b29c1f7 not found: ID does not exist" Oct 30 22:03:49.116350 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:49.116322 5748 scope.go:122] "RemoveContainer" containerID="1ddc6a01a5dc83bb1a2ff01f7c9f86bff070bc6be3f497c234534384fa0f0ab1" Oct 30 22:03:49.125697 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:49.125678 5748 scope.go:122] "RemoveContainer" containerID="1ddc6a01a5dc83bb1a2ff01f7c9f86bff070bc6be3f497c234534384fa0f0ab1" Oct 30 22:03:49.126023 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:03:49.126000 5748 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ddc6a01a5dc83bb1a2ff01f7c9f86bff070bc6be3f497c234534384fa0f0ab1\": container with ID starting with 1ddc6a01a5dc83bb1a2ff01f7c9f86bff070bc6be3f497c234534384fa0f0ab1 not found: ID does not exist" containerID="1ddc6a01a5dc83bb1a2ff01f7c9f86bff070bc6be3f497c234534384fa0f0ab1" Oct 30 22:03:49.126135 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:49.126025 5748 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ddc6a01a5dc83bb1a2ff01f7c9f86bff070bc6be3f497c234534384fa0f0ab1"} err="failed to get container status \"1ddc6a01a5dc83bb1a2ff01f7c9f86bff070bc6be3f497c234534384fa0f0ab1\": rpc error: code = NotFound desc = could not find container \"1ddc6a01a5dc83bb1a2ff01f7c9f86bff070bc6be3f497c234534384fa0f0ab1\": container with ID starting with 1ddc6a01a5dc83bb1a2ff01f7c9f86bff070bc6be3f497c234534384fa0f0ab1 not found: ID does not exist" Oct 30 22:03:49.261734 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:49.261705 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-6vzg4"] Oct 30 22:03:49.270628 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:49.270597 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-6vzg4"] Oct 30 22:03:49.379956 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:49.379873 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-ps4v9"] Oct 30 22:03:49.386266 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:49.386240 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-ps4v9"] Oct 30 22:03:49.505774 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:49.505738 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-cx2t5"] Oct 30 22:03:49.512111 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:49.512084 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-cx2t5"] Oct 30 22:03:50.417240 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:50.417204 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e9ffa82-546b-44a0-866b-44c89dfbb43e" path="/var/lib/kubelet/pods/0e9ffa82-546b-44a0-866b-44c89dfbb43e/volumes" Oct 30 22:03:50.417740 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:50.417724 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1346c154-7a8b-42a7-96b1-cf9996e38008" path="/var/lib/kubelet/pods/1346c154-7a8b-42a7-96b1-cf9996e38008/volumes" Oct 30 22:03:50.418244 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:03:50.418228 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53df3a33-7883-4793-b495-e5c7543432f0" path="/var/lib/kubelet/pods/53df3a33-7883-4793-b495-e5c7543432f0/volumes" Oct 30 22:04:00.147594 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:04:00.147552 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:04:02.904592 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:02.904556 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-fd2a94/test-model-fd2a94-kserve-678bdc5f47-h4hxc"] Oct 30 22:04:02.905128 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:02.905113 5748 cpu_manager.go:414] "containerMap: removing container" podUID="53df3a33-7883-4793-b495-e5c7543432f0" containerName="istio-proxy" Oct 30 22:04:02.905186 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:02.905128 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53df3a33-7883-4793-b495-e5c7543432f0" containerName="istio-proxy" Oct 30 22:04:02.905186 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:02.905145 5748 cpu_manager.go:414] "containerMap: removing container" podUID="0e9ffa82-546b-44a0-866b-44c89dfbb43e" containerName="istio-proxy" Oct 30 22:04:02.905186 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:02.905150 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e9ffa82-546b-44a0-866b-44c89dfbb43e" containerName="istio-proxy" Oct 30 22:04:02.905186 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:02.905157 5748 cpu_manager.go:414] "containerMap: removing container" podUID="1346c154-7a8b-42a7-96b1-cf9996e38008" containerName="istio-proxy" Oct 30 22:04:02.905186 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:02.905163 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1346c154-7a8b-42a7-96b1-cf9996e38008" containerName="istio-proxy" Oct 30 22:04:02.905738 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:02.905246 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="1346c154-7a8b-42a7-96b1-cf9996e38008" containerName="istio-proxy" Oct 30 22:04:02.905738 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:02.905255 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e9ffa82-546b-44a0-866b-44c89dfbb43e" containerName="istio-proxy" Oct 30 22:04:02.905738 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:02.905270 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="53df3a33-7883-4793-b495-e5c7543432f0" containerName="istio-proxy" Oct 30 22:04:02.908634 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:02.908616 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fd2a94/test-model-fd2a94-kserve-678bdc5f47-h4hxc" Oct 30 22:04:02.977761 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:02.977729 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-fd2a94/test-model-fd2a94-kserve-678bdc5f47-h4hxc"] Oct 30 22:04:03.001003 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:03.000970 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e428337c-c0ce-46b8-9471-96e53e5bc09c-dshm\") pod \"test-model-fd2a94-kserve-678bdc5f47-h4hxc\" (UID: \"e428337c-c0ce-46b8-9471-96e53e5bc09c\") " pod="ai-tenant-e2e-inf-a-fd2a94/test-model-fd2a94-kserve-678bdc5f47-h4hxc" Oct 30 22:04:03.001149 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:03.001028 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e428337c-c0ce-46b8-9471-96e53e5bc09c-tmp-dir\") pod \"test-model-fd2a94-kserve-678bdc5f47-h4hxc\" (UID: \"e428337c-c0ce-46b8-9471-96e53e5bc09c\") " pod="ai-tenant-e2e-inf-a-fd2a94/test-model-fd2a94-kserve-678bdc5f47-h4hxc" Oct 30 22:04:03.001149 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:03.001068 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e428337c-c0ce-46b8-9471-96e53e5bc09c-tls-certs\") pod \"test-model-fd2a94-kserve-678bdc5f47-h4hxc\" (UID: \"e428337c-c0ce-46b8-9471-96e53e5bc09c\") " pod="ai-tenant-e2e-inf-a-fd2a94/test-model-fd2a94-kserve-678bdc5f47-h4hxc" Oct 30 22:04:03.001149 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:03.001118 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e428337c-c0ce-46b8-9471-96e53e5bc09c-home\") pod \"test-model-fd2a94-kserve-678bdc5f47-h4hxc\" (UID: \"e428337c-c0ce-46b8-9471-96e53e5bc09c\") " pod="ai-tenant-e2e-inf-a-fd2a94/test-model-fd2a94-kserve-678bdc5f47-h4hxc" Oct 30 22:04:03.001149 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:03.001146 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e428337c-c0ce-46b8-9471-96e53e5bc09c-model-cache\") pod \"test-model-fd2a94-kserve-678bdc5f47-h4hxc\" (UID: \"e428337c-c0ce-46b8-9471-96e53e5bc09c\") " pod="ai-tenant-e2e-inf-a-fd2a94/test-model-fd2a94-kserve-678bdc5f47-h4hxc" Oct 30 22:04:03.001289 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:03.001176 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm6cl\" (UniqueName: \"kubernetes.io/projected/e428337c-c0ce-46b8-9471-96e53e5bc09c-kube-api-access-jm6cl\") pod \"test-model-fd2a94-kserve-678bdc5f47-h4hxc\" (UID: \"e428337c-c0ce-46b8-9471-96e53e5bc09c\") " pod="ai-tenant-e2e-inf-a-fd2a94/test-model-fd2a94-kserve-678bdc5f47-h4hxc" Oct 30 22:04:03.015052 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:03.015024 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-fd2a94\"/\"kube-root-ca.crt\"" Oct 30 22:04:03.015052 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:03.015038 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-fd2a94\"/\"default-dockercfg-jrzn9\"" Oct 30 22:04:03.015208 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:03.015062 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-fd2a94\"/\"openshift-service-ca.crt\"" Oct 30 22:04:03.015208 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:03.015108 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-fd2a94\"/\"test-model-fd2a94-kserve-self-signed-certs\"" Oct 30 22:04:03.102951 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:03.102880 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e428337c-c0ce-46b8-9471-96e53e5bc09c-tmp-dir\") pod \"test-model-fd2a94-kserve-678bdc5f47-h4hxc\" (UID: \"e428337c-c0ce-46b8-9471-96e53e5bc09c\") " pod="ai-tenant-e2e-inf-a-fd2a94/test-model-fd2a94-kserve-678bdc5f47-h4hxc" Oct 30 22:04:03.102951 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:03.102966 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e428337c-c0ce-46b8-9471-96e53e5bc09c-tls-certs\") pod \"test-model-fd2a94-kserve-678bdc5f47-h4hxc\" (UID: \"e428337c-c0ce-46b8-9471-96e53e5bc09c\") " pod="ai-tenant-e2e-inf-a-fd2a94/test-model-fd2a94-kserve-678bdc5f47-h4hxc" Oct 30 22:04:03.103236 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:03.103001 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e428337c-c0ce-46b8-9471-96e53e5bc09c-home\") pod \"test-model-fd2a94-kserve-678bdc5f47-h4hxc\" (UID: \"e428337c-c0ce-46b8-9471-96e53e5bc09c\") " pod="ai-tenant-e2e-inf-a-fd2a94/test-model-fd2a94-kserve-678bdc5f47-h4hxc" Oct 30 22:04:03.103236 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:03.103028 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e428337c-c0ce-46b8-9471-96e53e5bc09c-model-cache\") pod \"test-model-fd2a94-kserve-678bdc5f47-h4hxc\" (UID: \"e428337c-c0ce-46b8-9471-96e53e5bc09c\") " pod="ai-tenant-e2e-inf-a-fd2a94/test-model-fd2a94-kserve-678bdc5f47-h4hxc" Oct 30 22:04:03.103236 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:03.103080 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm6cl\" (UniqueName: \"kubernetes.io/projected/e428337c-c0ce-46b8-9471-96e53e5bc09c-kube-api-access-jm6cl\") pod \"test-model-fd2a94-kserve-678bdc5f47-h4hxc\" (UID: \"e428337c-c0ce-46b8-9471-96e53e5bc09c\") " pod="ai-tenant-e2e-inf-a-fd2a94/test-model-fd2a94-kserve-678bdc5f47-h4hxc" Oct 30 22:04:03.103421 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:03.103286 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e428337c-c0ce-46b8-9471-96e53e5bc09c-dshm\") pod \"test-model-fd2a94-kserve-678bdc5f47-h4hxc\" (UID: \"e428337c-c0ce-46b8-9471-96e53e5bc09c\") " pod="ai-tenant-e2e-inf-a-fd2a94/test-model-fd2a94-kserve-678bdc5f47-h4hxc" Oct 30 22:04:03.103421 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:03.103307 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e428337c-c0ce-46b8-9471-96e53e5bc09c-tmp-dir\") pod \"test-model-fd2a94-kserve-678bdc5f47-h4hxc\" (UID: \"e428337c-c0ce-46b8-9471-96e53e5bc09c\") " pod="ai-tenant-e2e-inf-a-fd2a94/test-model-fd2a94-kserve-678bdc5f47-h4hxc" Oct 30 22:04:03.103421 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:03.103406 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e428337c-c0ce-46b8-9471-96e53e5bc09c-home\") pod \"test-model-fd2a94-kserve-678bdc5f47-h4hxc\" (UID: \"e428337c-c0ce-46b8-9471-96e53e5bc09c\") " pod="ai-tenant-e2e-inf-a-fd2a94/test-model-fd2a94-kserve-678bdc5f47-h4hxc" Oct 30 22:04:03.103594 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:03.103453 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e428337c-c0ce-46b8-9471-96e53e5bc09c-model-cache\") pod \"test-model-fd2a94-kserve-678bdc5f47-h4hxc\" (UID: \"e428337c-c0ce-46b8-9471-96e53e5bc09c\") " pod="ai-tenant-e2e-inf-a-fd2a94/test-model-fd2a94-kserve-678bdc5f47-h4hxc" Oct 30 22:04:03.105999 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:03.105972 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e428337c-c0ce-46b8-9471-96e53e5bc09c-dshm\") pod \"test-model-fd2a94-kserve-678bdc5f47-h4hxc\" (UID: \"e428337c-c0ce-46b8-9471-96e53e5bc09c\") " pod="ai-tenant-e2e-inf-a-fd2a94/test-model-fd2a94-kserve-678bdc5f47-h4hxc" Oct 30 22:04:03.107789 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:03.107765 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e428337c-c0ce-46b8-9471-96e53e5bc09c-tls-certs\") pod \"test-model-fd2a94-kserve-678bdc5f47-h4hxc\" (UID: \"e428337c-c0ce-46b8-9471-96e53e5bc09c\") " pod="ai-tenant-e2e-inf-a-fd2a94/test-model-fd2a94-kserve-678bdc5f47-h4hxc" Oct 30 22:04:03.168061 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:03.167995 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm6cl\" (UniqueName: \"kubernetes.io/projected/e428337c-c0ce-46b8-9471-96e53e5bc09c-kube-api-access-jm6cl\") pod \"test-model-fd2a94-kserve-678bdc5f47-h4hxc\" (UID: \"e428337c-c0ce-46b8-9471-96e53e5bc09c\") " pod="ai-tenant-e2e-inf-a-fd2a94/test-model-fd2a94-kserve-678bdc5f47-h4hxc" Oct 30 22:04:03.230454 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:03.230422 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fd2a94/test-model-fd2a94-kserve-678bdc5f47-h4hxc" Oct 30 22:04:03.613450 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:03.613425 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-fd2a94/test-model-fd2a94-kserve-678bdc5f47-h4hxc"] Oct 30 22:04:03.615708 ip-10-0-143-61 kubenswrapper[5748]: W1030 22:04:03.615680 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode428337c_c0ce_46b8_9471_96e53e5bc09c.slice/crio-b29b3062e88d0ce1b50f34e16c05887b89f8e98f6c44463573d916bb641bf6e8 WatchSource:0}: Error finding container b29b3062e88d0ce1b50f34e16c05887b89f8e98f6c44463573d916bb641bf6e8: Status 404 returned error can't find the container with id b29b3062e88d0ce1b50f34e16c05887b89f8e98f6c44463573d916bb641bf6e8 Oct 30 22:04:04.200347 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:04.200307 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fd2a94/test-model-fd2a94-kserve-678bdc5f47-h4hxc" event={"ID":"e428337c-c0ce-46b8-9471-96e53e5bc09c","Type":"ContainerStarted","Data":"2573c134e47f3366893f064af88bd18c9b842564fbc627cfa6b8e2e99ae3af82"} Oct 30 22:04:04.200347 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:04.200352 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fd2a94/test-model-fd2a94-kserve-678bdc5f47-h4hxc" event={"ID":"e428337c-c0ce-46b8-9471-96e53e5bc09c","Type":"ContainerStarted","Data":"b29b3062e88d0ce1b50f34e16c05887b89f8e98f6c44463573d916bb641bf6e8"} Oct 30 22:04:04.230668 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:04.230642 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-fd2a94/test-model-fd2a94-kserve-678bdc5f47-h4hxc" Oct 30 22:04:04.320301 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:04.320235 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-fd2a94/test-model-fd2a94-kserve-678bdc5f47-h4hxc" podStartSLOduration=2.320219638 podStartE2EDuration="2.320219638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:04.319117245 +0000 UTC m=+2428.061638179" watchObservedRunningTime="2026-10-30 22:04:04.320219638 +0000 UTC m=+2428.062740574" Oct 30 22:04:10.221289 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:10.221249 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh"] Oct 30 22:04:10.221819 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:10.221450 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh" podUID="b6c9d2bc-60d6-494c-af57-574e85c4861c" containerName="istio-proxy" containerID="cri-o://a88051189ba8ed7703e8bcb3da5b8a155ae851a3fa5b21c33fe875d27363b58c" gracePeriod=30 Oct 30 22:04:13.235642 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:13.235606 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-fd2a94/test-model-fd2a94-kserve-678bdc5f47-h4hxc" Oct 30 22:04:13.280791 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:13.280661 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-fd2a94/test-model-fd2a94-kserve-678bdc5f47-h4hxc" Oct 30 22:04:13.335687 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:13.335147 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-fd2a94/test-model-fd2a94-kserve-678bdc5f47-h4hxc" Oct 30 22:04:13.803987 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:13.803955 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-91874e-p4jlh"] Oct 30 22:04:13.808746 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:13.808720 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-91874e-p4jlh"] Oct 30 22:04:14.416974 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:14.416941 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbabbf0d-88f2-4775-ab15-44a80c63d31d" path="/var/lib/kubelet/pods/fbabbf0d-88f2-4775-ab15-44a80c63d31d/volumes" Oct 30 22:04:14.680914 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:14.680819 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz"] Oct 30 22:04:14.681102 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:14.681079 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz" podUID="b659bb43-102b-4936-87ed-d1b955262349" containerName="istio-proxy" containerID="cri-o://72169f1df2df2264c2602a3e930e762511bb1882bfa9eed5e62397e130283724" gracePeriod=30 Oct 30 22:04:15.504317 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:15.504256 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh" Oct 30 22:04:15.555224 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:15.555200 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6c9d2bc-60d6-494c-af57-574e85c4861c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6c9d2bc-60d6-494c-af57-574e85c4861c-istiod-ca-cert\") pod \"b6c9d2bc-60d6-494c-af57-574e85c4861c\" (UID: \"b6c9d2bc-60d6-494c-af57-574e85c4861c\") " Oct 30 22:04:15.555347 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:15.555272 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6c9d2bc-60d6-494c-af57-574e85c4861c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6c9d2bc-60d6-494c-af57-574e85c4861c-credential-socket\") pod \"b6c9d2bc-60d6-494c-af57-574e85c4861c\" (UID: \"b6c9d2bc-60d6-494c-af57-574e85c4861c\") " Oct 30 22:04:15.555347 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:15.555303 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6c9d2bc-60d6-494c-af57-574e85c4861c-kube-api-access-tqjhf\" (UniqueName: \"kubernetes.io/projected/b6c9d2bc-60d6-494c-af57-574e85c4861c-kube-api-access-tqjhf\") pod \"b6c9d2bc-60d6-494c-af57-574e85c4861c\" (UID: \"b6c9d2bc-60d6-494c-af57-574e85c4861c\") " Oct 30 22:04:15.555347 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:15.555325 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6c9d2bc-60d6-494c-af57-574e85c4861c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6c9d2bc-60d6-494c-af57-574e85c4861c-workload-certs\") pod \"b6c9d2bc-60d6-494c-af57-574e85c4861c\" (UID: \"b6c9d2bc-60d6-494c-af57-574e85c4861c\") " Oct 30 22:04:15.555455 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:15.555355 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6c9d2bc-60d6-494c-af57-574e85c4861c-istio-token\" (UniqueName: \"kubernetes.io/projected/b6c9d2bc-60d6-494c-af57-574e85c4861c-istio-token\") pod \"b6c9d2bc-60d6-494c-af57-574e85c4861c\" (UID: \"b6c9d2bc-60d6-494c-af57-574e85c4861c\") " Oct 30 22:04:15.555508 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:15.555481 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6c9d2bc-60d6-494c-af57-574e85c4861c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6c9d2bc-60d6-494c-af57-574e85c4861c-istio-data\") pod \"b6c9d2bc-60d6-494c-af57-574e85c4861c\" (UID: \"b6c9d2bc-60d6-494c-af57-574e85c4861c\") " Oct 30 22:04:15.555649 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:15.555566 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6c9d2bc-60d6-494c-af57-574e85c4861c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6c9d2bc-60d6-494c-af57-574e85c4861c-istio-envoy\") pod \"b6c9d2bc-60d6-494c-af57-574e85c4861c\" (UID: \"b6c9d2bc-60d6-494c-af57-574e85c4861c\") " Oct 30 22:04:15.555649 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:15.555592 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6c9d2bc-60d6-494c-af57-574e85c4861c-credential-socket" pod "b6c9d2bc-60d6-494c-af57-574e85c4861c" (UID: "b6c9d2bc-60d6-494c-af57-574e85c4861c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:15.555649 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:15.555615 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6c9d2bc-60d6-494c-af57-574e85c4861c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6c9d2bc-60d6-494c-af57-574e85c4861c-workload-socket\") pod \"b6c9d2bc-60d6-494c-af57-574e85c4861c\" (UID: \"b6c9d2bc-60d6-494c-af57-574e85c4861c\") " Oct 30 22:04:15.555832 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:15.555731 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6c9d2bc-60d6-494c-af57-574e85c4861c-workload-certs" pod "b6c9d2bc-60d6-494c-af57-574e85c4861c" (UID: "b6c9d2bc-60d6-494c-af57-574e85c4861c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:15.555832 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:15.555743 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6c9d2bc-60d6-494c-af57-574e85c4861c-istio-data" pod "b6c9d2bc-60d6-494c-af57-574e85c4861c" (UID: "b6c9d2bc-60d6-494c-af57-574e85c4861c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:15.555832 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:15.555597 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6c9d2bc-60d6-494c-af57-574e85c4861c-istiod-ca-cert" pod "b6c9d2bc-60d6-494c-af57-574e85c4861c" (UID: "b6c9d2bc-60d6-494c-af57-574e85c4861c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:15.555832 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:15.555763 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b6c9d2bc-60d6-494c-af57-574e85c4861c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6c9d2bc-60d6-494c-af57-574e85c4861c-istio-podinfo\") pod \"b6c9d2bc-60d6-494c-af57-574e85c4861c\" (UID: \"b6c9d2bc-60d6-494c-af57-574e85c4861c\") " Oct 30 22:04:15.556096 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:15.555939 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6c9d2bc-60d6-494c-af57-574e85c4861c-workload-socket" pod "b6c9d2bc-60d6-494c-af57-574e85c4861c" (UID: "b6c9d2bc-60d6-494c-af57-574e85c4861c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:15.556379 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:15.556357 5748 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6c9d2bc-60d6-494c-af57-574e85c4861c-workload-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:15.556439 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:15.556387 5748 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6c9d2bc-60d6-494c-af57-574e85c4861c-istiod-ca-cert\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:15.556439 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:15.556404 5748 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6c9d2bc-60d6-494c-af57-574e85c4861c-credential-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:15.556439 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:15.556419 5748 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6c9d2bc-60d6-494c-af57-574e85c4861c-workload-certs\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:15.556439 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:15.556432 5748 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6c9d2bc-60d6-494c-af57-574e85c4861c-istio-data\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:15.557690 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:15.557666 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6c9d2bc-60d6-494c-af57-574e85c4861c-istio-envoy" pod "b6c9d2bc-60d6-494c-af57-574e85c4861c" (UID: "b6c9d2bc-60d6-494c-af57-574e85c4861c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:15.557867 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:15.557845 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6c9d2bc-60d6-494c-af57-574e85c4861c-istio-token" pod "b6c9d2bc-60d6-494c-af57-574e85c4861c" (UID: "b6c9d2bc-60d6-494c-af57-574e85c4861c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:15.558135 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:15.558117 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6c9d2bc-60d6-494c-af57-574e85c4861c-kube-api-access-tqjhf" pod "b6c9d2bc-60d6-494c-af57-574e85c4861c" (UID: "b6c9d2bc-60d6-494c-af57-574e85c4861c"). InnerVolumeSpecName "kube-api-access-tqjhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:15.558195 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:15.558117 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b6c9d2bc-60d6-494c-af57-574e85c4861c-istio-podinfo" pod "b6c9d2bc-60d6-494c-af57-574e85c4861c" (UID: "b6c9d2bc-60d6-494c-af57-574e85c4861c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:04:15.657393 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:15.657360 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqjhf\" (UniqueName: \"kubernetes.io/projected/b6c9d2bc-60d6-494c-af57-574e85c4861c-kube-api-access-tqjhf\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:15.657393 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:15.657389 5748 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6c9d2bc-60d6-494c-af57-574e85c4861c-istio-token\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:15.657586 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:15.657398 5748 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6c9d2bc-60d6-494c-af57-574e85c4861c-istio-envoy\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:15.657586 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:15.657420 5748 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6c9d2bc-60d6-494c-af57-574e85c4861c-istio-podinfo\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:16.315289 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:16.315251 5748 generic.go:356] "Generic (PLEG): container finished" podID="b6c9d2bc-60d6-494c-af57-574e85c4861c" containerID="a88051189ba8ed7703e8bcb3da5b8a155ae851a3fa5b21c33fe875d27363b58c" exitCode=0 Oct 30 22:04:16.315471 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:16.315297 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh" Oct 30 22:04:16.315471 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:16.315326 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh" event={"ID":"b6c9d2bc-60d6-494c-af57-574e85c4861c","Type":"ContainerDied","Data":"a88051189ba8ed7703e8bcb3da5b8a155ae851a3fa5b21c33fe875d27363b58c"} Oct 30 22:04:16.315471 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:16.315384 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh" event={"ID":"b6c9d2bc-60d6-494c-af57-574e85c4861c","Type":"ContainerDied","Data":"7c3728c4fc5cc1f1d88636c20ecf018aa89ee9bb2963f4449e9212fc01dd3178"} Oct 30 22:04:16.315471 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:16.315409 5748 scope.go:122] "RemoveContainer" containerID="a88051189ba8ed7703e8bcb3da5b8a155ae851a3fa5b21c33fe875d27363b58c" Oct 30 22:04:16.326651 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:16.326631 5748 scope.go:122] "RemoveContainer" containerID="a88051189ba8ed7703e8bcb3da5b8a155ae851a3fa5b21c33fe875d27363b58c" Oct 30 22:04:16.326947 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:04:16.326926 5748 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a88051189ba8ed7703e8bcb3da5b8a155ae851a3fa5b21c33fe875d27363b58c\": container with ID starting with a88051189ba8ed7703e8bcb3da5b8a155ae851a3fa5b21c33fe875d27363b58c not found: ID does not exist" containerID="a88051189ba8ed7703e8bcb3da5b8a155ae851a3fa5b21c33fe875d27363b58c" Oct 30 22:04:16.327024 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:16.326960 5748 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a88051189ba8ed7703e8bcb3da5b8a155ae851a3fa5b21c33fe875d27363b58c"} err="failed to get container status \"a88051189ba8ed7703e8bcb3da5b8a155ae851a3fa5b21c33fe875d27363b58c\": rpc error: code = NotFound desc = could not find container \"a88051189ba8ed7703e8bcb3da5b8a155ae851a3fa5b21c33fe875d27363b58c\": container with ID starting with a88051189ba8ed7703e8bcb3da5b8a155ae851a3fa5b21c33fe875d27363b58c not found: ID does not exist" Oct 30 22:04:16.483615 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:16.483582 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh"] Oct 30 22:04:16.534069 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:16.534038 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-bfrbh"] Oct 30 22:04:18.416753 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:18.416717 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6c9d2bc-60d6-494c-af57-574e85c4861c" path="/var/lib/kubelet/pods/b6c9d2bc-60d6-494c-af57-574e85c4861c/volumes" Oct 30 22:04:19.949975 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:19.949922 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz" Oct 30 22:04:20.006064 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:20.006029 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b659bb43-102b-4936-87ed-d1b955262349-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b659bb43-102b-4936-87ed-d1b955262349-workload-certs\") pod \"b659bb43-102b-4936-87ed-d1b955262349\" (UID: \"b659bb43-102b-4936-87ed-d1b955262349\") " Oct 30 22:04:20.006230 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:20.006106 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b659bb43-102b-4936-87ed-d1b955262349-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b659bb43-102b-4936-87ed-d1b955262349-workload-socket\") pod \"b659bb43-102b-4936-87ed-d1b955262349\" (UID: \"b659bb43-102b-4936-87ed-d1b955262349\") " Oct 30 22:04:20.006230 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:20.006131 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b659bb43-102b-4936-87ed-d1b955262349-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b659bb43-102b-4936-87ed-d1b955262349-istio-data\") pod \"b659bb43-102b-4936-87ed-d1b955262349\" (UID: \"b659bb43-102b-4936-87ed-d1b955262349\") " Oct 30 22:04:20.006230 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:20.006166 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b659bb43-102b-4936-87ed-d1b955262349-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b659bb43-102b-4936-87ed-d1b955262349-istio-envoy\") pod \"b659bb43-102b-4936-87ed-d1b955262349\" (UID: \"b659bb43-102b-4936-87ed-d1b955262349\") " Oct 30 22:04:20.006230 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:20.006229 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b659bb43-102b-4936-87ed-d1b955262349-kube-api-access-tzqwk\" (UniqueName: \"kubernetes.io/projected/b659bb43-102b-4936-87ed-d1b955262349-kube-api-access-tzqwk\") pod \"b659bb43-102b-4936-87ed-d1b955262349\" (UID: \"b659bb43-102b-4936-87ed-d1b955262349\") " Oct 30 22:04:20.006478 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:20.006256 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b659bb43-102b-4936-87ed-d1b955262349-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b659bb43-102b-4936-87ed-d1b955262349-credential-socket\") pod \"b659bb43-102b-4936-87ed-d1b955262349\" (UID: \"b659bb43-102b-4936-87ed-d1b955262349\") " Oct 30 22:04:20.006478 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:20.006282 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b659bb43-102b-4936-87ed-d1b955262349-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b659bb43-102b-4936-87ed-d1b955262349-istiod-ca-cert\") pod \"b659bb43-102b-4936-87ed-d1b955262349\" (UID: \"b659bb43-102b-4936-87ed-d1b955262349\") " Oct 30 22:04:20.006478 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:20.006336 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b659bb43-102b-4936-87ed-d1b955262349-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b659bb43-102b-4936-87ed-d1b955262349-istio-podinfo\") pod \"b659bb43-102b-4936-87ed-d1b955262349\" (UID: \"b659bb43-102b-4936-87ed-d1b955262349\") " Oct 30 22:04:20.006478 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:20.006439 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b659bb43-102b-4936-87ed-d1b955262349-istio-token\" (UniqueName: \"kubernetes.io/projected/b659bb43-102b-4936-87ed-d1b955262349-istio-token\") pod \"b659bb43-102b-4936-87ed-d1b955262349\" (UID: \"b659bb43-102b-4936-87ed-d1b955262349\") " Oct 30 22:04:20.006478 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:20.006331 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b659bb43-102b-4936-87ed-d1b955262349-workload-certs" pod "b659bb43-102b-4936-87ed-d1b955262349" (UID: "b659bb43-102b-4936-87ed-d1b955262349"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:20.006755 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:20.006447 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b659bb43-102b-4936-87ed-d1b955262349-workload-socket" pod "b659bb43-102b-4936-87ed-d1b955262349" (UID: "b659bb43-102b-4936-87ed-d1b955262349"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:20.006755 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:20.006459 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b659bb43-102b-4936-87ed-d1b955262349-istio-data" pod "b659bb43-102b-4936-87ed-d1b955262349" (UID: "b659bb43-102b-4936-87ed-d1b955262349"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:20.006755 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:20.006590 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b659bb43-102b-4936-87ed-d1b955262349-credential-socket" pod "b659bb43-102b-4936-87ed-d1b955262349" (UID: "b659bb43-102b-4936-87ed-d1b955262349"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:20.006755 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:20.006693 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b659bb43-102b-4936-87ed-d1b955262349-istiod-ca-cert" pod "b659bb43-102b-4936-87ed-d1b955262349" (UID: "b659bb43-102b-4936-87ed-d1b955262349"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:20.007097 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:20.007057 5748 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b659bb43-102b-4936-87ed-d1b955262349-credential-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:20.007097 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:20.007085 5748 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b659bb43-102b-4936-87ed-d1b955262349-istiod-ca-cert\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:20.007242 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:20.007100 5748 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b659bb43-102b-4936-87ed-d1b955262349-workload-certs\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:20.007242 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:20.007116 5748 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b659bb43-102b-4936-87ed-d1b955262349-workload-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:20.007242 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:20.007129 5748 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b659bb43-102b-4936-87ed-d1b955262349-istio-data\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:20.008523 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:20.008503 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b659bb43-102b-4936-87ed-d1b955262349-istio-envoy" pod "b659bb43-102b-4936-87ed-d1b955262349" (UID: "b659bb43-102b-4936-87ed-d1b955262349"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:20.008606 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:20.008555 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b659bb43-102b-4936-87ed-d1b955262349-istio-podinfo" pod "b659bb43-102b-4936-87ed-d1b955262349" (UID: "b659bb43-102b-4936-87ed-d1b955262349"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:04:20.008851 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:20.008832 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b659bb43-102b-4936-87ed-d1b955262349-kube-api-access-tzqwk" pod "b659bb43-102b-4936-87ed-d1b955262349" (UID: "b659bb43-102b-4936-87ed-d1b955262349"). InnerVolumeSpecName "kube-api-access-tzqwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:20.008970 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:20.008855 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b659bb43-102b-4936-87ed-d1b955262349-istio-token" pod "b659bb43-102b-4936-87ed-d1b955262349" (UID: "b659bb43-102b-4936-87ed-d1b955262349"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:20.108188 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:20.108153 5748 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b659bb43-102b-4936-87ed-d1b955262349-istio-podinfo\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:20.108188 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:20.108183 5748 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b659bb43-102b-4936-87ed-d1b955262349-istio-token\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:20.108372 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:20.108197 5748 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b659bb43-102b-4936-87ed-d1b955262349-istio-envoy\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:20.108372 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:20.108209 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzqwk\" (UniqueName: \"kubernetes.io/projected/b659bb43-102b-4936-87ed-d1b955262349-kube-api-access-tzqwk\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:20.344611 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:20.344519 5748 generic.go:356] "Generic (PLEG): container finished" podID="b659bb43-102b-4936-87ed-d1b955262349" containerID="72169f1df2df2264c2602a3e930e762511bb1882bfa9eed5e62397e130283724" exitCode=0 Oct 30 22:04:20.344611 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:20.344572 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz" Oct 30 22:04:20.344798 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:20.344610 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz" event={"ID":"b659bb43-102b-4936-87ed-d1b955262349","Type":"ContainerDied","Data":"72169f1df2df2264c2602a3e930e762511bb1882bfa9eed5e62397e130283724"} Oct 30 22:04:20.344798 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:20.344686 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz" event={"ID":"b659bb43-102b-4936-87ed-d1b955262349","Type":"ContainerDied","Data":"c66447f107a9863602b7b7984266d253f9ad6498663d753c059a2670cd62b10d"} Oct 30 22:04:20.344798 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:20.344709 5748 scope.go:122] "RemoveContainer" containerID="72169f1df2df2264c2602a3e930e762511bb1882bfa9eed5e62397e130283724" Oct 30 22:04:20.355481 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:20.355461 5748 scope.go:122] "RemoveContainer" containerID="72169f1df2df2264c2602a3e930e762511bb1882bfa9eed5e62397e130283724" Oct 30 22:04:20.355739 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:04:20.355721 5748 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72169f1df2df2264c2602a3e930e762511bb1882bfa9eed5e62397e130283724\": container with ID starting with 72169f1df2df2264c2602a3e930e762511bb1882bfa9eed5e62397e130283724 not found: ID does not exist" containerID="72169f1df2df2264c2602a3e930e762511bb1882bfa9eed5e62397e130283724" Oct 30 22:04:20.355789 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:20.355747 5748 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72169f1df2df2264c2602a3e930e762511bb1882bfa9eed5e62397e130283724"} err="failed to get container status \"72169f1df2df2264c2602a3e930e762511bb1882bfa9eed5e62397e130283724\": rpc error: code = NotFound desc = could not find container \"72169f1df2df2264c2602a3e930e762511bb1882bfa9eed5e62397e130283724\": container with ID starting with 72169f1df2df2264c2602a3e930e762511bb1882bfa9eed5e62397e130283724 not found: ID does not exist" Oct 30 22:04:20.512057 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:20.512026 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz"] Oct 30 22:04:20.544299 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:20.544264 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-zc5xz"] Oct 30 22:04:21.151295 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:21.151257 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7"] Oct 30 22:04:21.151801 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:21.151765 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7" podUID="d2cdb6d5-0628-435d-ade1-e0aa84440718" containerName="istio-proxy" containerID="cri-o://2caff4996f2261f9c21a7bb1a193e9df5906df30f09c82b14ecfef46f90dcd86" gracePeriod=30 Oct 30 22:04:21.901067 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:21.901034 5748 patch_prober.go:29] interesting pod/e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:04:21.901249 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:21.901088 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7" podUID="d2cdb6d5-0628-435d-ade1-e0aa84440718" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:04:22.424546 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:22.424504 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b659bb43-102b-4936-87ed-d1b955262349" path="/var/lib/kubelet/pods/b659bb43-102b-4936-87ed-d1b955262349/volumes" Oct 30 22:04:26.390917 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:26.390860 5748 generic.go:356] "Generic (PLEG): container finished" podID="d2cdb6d5-0628-435d-ade1-e0aa84440718" containerID="2caff4996f2261f9c21a7bb1a193e9df5906df30f09c82b14ecfef46f90dcd86" exitCode=0 Oct 30 22:04:26.391285 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:26.390934 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7" event={"ID":"d2cdb6d5-0628-435d-ade1-e0aa84440718","Type":"ContainerDied","Data":"2caff4996f2261f9c21a7bb1a193e9df5906df30f09c82b14ecfef46f90dcd86"} Oct 30 22:04:26.420659 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:26.420595 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7" Oct 30 22:04:26.587979 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:26.587942 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2cdb6d5-0628-435d-ade1-e0aa84440718-istio-token\" (UniqueName: \"kubernetes.io/projected/d2cdb6d5-0628-435d-ade1-e0aa84440718-istio-token\") pod \"d2cdb6d5-0628-435d-ade1-e0aa84440718\" (UID: \"d2cdb6d5-0628-435d-ade1-e0aa84440718\") " Oct 30 22:04:26.588148 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:26.588047 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2cdb6d5-0628-435d-ade1-e0aa84440718-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2cdb6d5-0628-435d-ade1-e0aa84440718-workload-certs\") pod \"d2cdb6d5-0628-435d-ade1-e0aa84440718\" (UID: \"d2cdb6d5-0628-435d-ade1-e0aa84440718\") " Oct 30 22:04:26.588148 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:26.588072 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2cdb6d5-0628-435d-ade1-e0aa84440718-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2cdb6d5-0628-435d-ade1-e0aa84440718-istiod-ca-cert\") pod \"d2cdb6d5-0628-435d-ade1-e0aa84440718\" (UID: \"d2cdb6d5-0628-435d-ade1-e0aa84440718\") " Oct 30 22:04:26.588148 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:26.588119 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2cdb6d5-0628-435d-ade1-e0aa84440718-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2cdb6d5-0628-435d-ade1-e0aa84440718-credential-socket\") pod \"d2cdb6d5-0628-435d-ade1-e0aa84440718\" (UID: \"d2cdb6d5-0628-435d-ade1-e0aa84440718\") " Oct 30 22:04:26.588148 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:26.588142 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2cdb6d5-0628-435d-ade1-e0aa84440718-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2cdb6d5-0628-435d-ade1-e0aa84440718-istio-data\") pod \"d2cdb6d5-0628-435d-ade1-e0aa84440718\" (UID: \"d2cdb6d5-0628-435d-ade1-e0aa84440718\") " Oct 30 22:04:26.588434 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:26.588178 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2cdb6d5-0628-435d-ade1-e0aa84440718-kube-api-access-c87t2\" (UniqueName: \"kubernetes.io/projected/d2cdb6d5-0628-435d-ade1-e0aa84440718-kube-api-access-c87t2\") pod \"d2cdb6d5-0628-435d-ade1-e0aa84440718\" (UID: \"d2cdb6d5-0628-435d-ade1-e0aa84440718\") " Oct 30 22:04:26.588434 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:26.588207 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2cdb6d5-0628-435d-ade1-e0aa84440718-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2cdb6d5-0628-435d-ade1-e0aa84440718-workload-socket\") pod \"d2cdb6d5-0628-435d-ade1-e0aa84440718\" (UID: \"d2cdb6d5-0628-435d-ade1-e0aa84440718\") " Oct 30 22:04:26.588434 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:26.588279 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d2cdb6d5-0628-435d-ade1-e0aa84440718-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2cdb6d5-0628-435d-ade1-e0aa84440718-istio-podinfo\") pod \"d2cdb6d5-0628-435d-ade1-e0aa84440718\" (UID: \"d2cdb6d5-0628-435d-ade1-e0aa84440718\") " Oct 30 22:04:26.588434 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:26.588309 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2cdb6d5-0628-435d-ade1-e0aa84440718-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2cdb6d5-0628-435d-ade1-e0aa84440718-istio-envoy\") pod \"d2cdb6d5-0628-435d-ade1-e0aa84440718\" (UID: \"d2cdb6d5-0628-435d-ade1-e0aa84440718\") " Oct 30 22:04:26.588434 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:26.588368 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2cdb6d5-0628-435d-ade1-e0aa84440718-workload-certs" pod "d2cdb6d5-0628-435d-ade1-e0aa84440718" (UID: "d2cdb6d5-0628-435d-ade1-e0aa84440718"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:26.588709 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:26.588458 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2cdb6d5-0628-435d-ade1-e0aa84440718-istio-data" pod "d2cdb6d5-0628-435d-ade1-e0aa84440718" (UID: "d2cdb6d5-0628-435d-ade1-e0aa84440718"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:26.588709 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:26.588603 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2cdb6d5-0628-435d-ade1-e0aa84440718-workload-socket" pod "d2cdb6d5-0628-435d-ade1-e0aa84440718" (UID: "d2cdb6d5-0628-435d-ade1-e0aa84440718"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:26.588709 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:26.588529 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2cdb6d5-0628-435d-ade1-e0aa84440718-istiod-ca-cert" pod "d2cdb6d5-0628-435d-ade1-e0aa84440718" (UID: "d2cdb6d5-0628-435d-ade1-e0aa84440718"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:26.588709 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:26.588678 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2cdb6d5-0628-435d-ade1-e0aa84440718-credential-socket" pod "d2cdb6d5-0628-435d-ade1-e0aa84440718" (UID: "d2cdb6d5-0628-435d-ade1-e0aa84440718"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:26.588975 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:26.588956 5748 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2cdb6d5-0628-435d-ade1-e0aa84440718-workload-certs\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:26.589035 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:26.588981 5748 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2cdb6d5-0628-435d-ade1-e0aa84440718-istiod-ca-cert\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:26.589035 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:26.588997 5748 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2cdb6d5-0628-435d-ade1-e0aa84440718-credential-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:26.589035 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:26.589010 5748 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2cdb6d5-0628-435d-ade1-e0aa84440718-istio-data\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:26.589035 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:26.589023 5748 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2cdb6d5-0628-435d-ade1-e0aa84440718-workload-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:26.590478 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:26.590449 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2cdb6d5-0628-435d-ade1-e0aa84440718-istio-envoy" pod "d2cdb6d5-0628-435d-ade1-e0aa84440718" (UID: "d2cdb6d5-0628-435d-ade1-e0aa84440718"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:26.590575 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:26.590489 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2cdb6d5-0628-435d-ade1-e0aa84440718-kube-api-access-c87t2" pod "d2cdb6d5-0628-435d-ade1-e0aa84440718" (UID: "d2cdb6d5-0628-435d-ade1-e0aa84440718"). InnerVolumeSpecName "kube-api-access-c87t2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:26.590659 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:26.590641 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2cdb6d5-0628-435d-ade1-e0aa84440718-istio-token" pod "d2cdb6d5-0628-435d-ade1-e0aa84440718" (UID: "d2cdb6d5-0628-435d-ade1-e0aa84440718"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:26.590699 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:26.590681 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2cdb6d5-0628-435d-ade1-e0aa84440718-istio-podinfo" pod "d2cdb6d5-0628-435d-ade1-e0aa84440718" (UID: "d2cdb6d5-0628-435d-ade1-e0aa84440718"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:04:26.690076 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:26.690013 5748 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2cdb6d5-0628-435d-ade1-e0aa84440718-istio-podinfo\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:26.690076 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:26.690037 5748 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2cdb6d5-0628-435d-ade1-e0aa84440718-istio-envoy\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:26.690076 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:26.690047 5748 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2cdb6d5-0628-435d-ade1-e0aa84440718-istio-token\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:26.690076 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:26.690056 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c87t2\" (UniqueName: \"kubernetes.io/projected/d2cdb6d5-0628-435d-ade1-e0aa84440718-kube-api-access-c87t2\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:27.399959 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:27.399926 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7" Oct 30 22:04:27.400385 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:27.399926 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7" event={"ID":"d2cdb6d5-0628-435d-ade1-e0aa84440718","Type":"ContainerDied","Data":"e36ab17ca70ef7ee51e8967050226d15bd689b65675e10e6a101ea913dde862b"} Oct 30 22:04:27.400385 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:27.400065 5748 scope.go:122] "RemoveContainer" containerID="2caff4996f2261f9c21a7bb1a193e9df5906df30f09c82b14ecfef46f90dcd86" Oct 30 22:04:27.617994 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:27.617962 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7"] Oct 30 22:04:27.622746 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:27.622716 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-2q5s7"] Oct 30 22:04:28.416885 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:28.416841 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2cdb6d5-0628-435d-ade1-e0aa84440718" path="/var/lib/kubelet/pods/d2cdb6d5-0628-435d-ade1-e0aa84440718/volumes" Oct 30 22:04:30.145699 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:04:30.145657 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:04:38.052767 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:38.052732 5748 scope.go:122] "RemoveContainer" containerID="8631fd647f8cae67c2ffd457b79a187237484b2eaeef9122095c7b097b554a69" Oct 30 22:04:38.061961 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:38.061940 5748 scope.go:122] "RemoveContainer" containerID="d81d9e5274309ca1cd2101a5f2c5a668aa9c87ca31ffdb433438ddd7ed40da75" Oct 30 22:04:38.070689 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:38.070663 5748 scope.go:122] "RemoveContainer" containerID="0b65111ae775092ffedb1c67b2d751164891386664fa1c78056c6ac7fe0a99ec" Oct 30 22:04:51.577473 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:51.577443 5748 generic.go:356] "Generic (PLEG): container finished" podID="024f007a-0f46-4093-8554-93a335039f10" containerID="6f596f846951d9760f05100004a2df05e925bb0ac478a90729c4d3a45d7c021a" exitCode=0 Oct 30 22:04:51.577917 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:51.577511 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v" event={"ID":"024f007a-0f46-4093-8554-93a335039f10","Type":"ContainerDied","Data":"6f596f846951d9760f05100004a2df05e925bb0ac478a90729c4d3a45d7c021a"} Oct 30 22:04:51.577982 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:51.577966 5748 scope.go:122] "RemoveContainer" containerID="6f596f846951d9760f05100004a2df05e925bb0ac478a90729c4d3a45d7c021a" Oct 30 22:04:51.579593 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:51.579563 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:04:51.579660 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:51.579635 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:04:51.579700 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:51.579667 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:04:51.962007 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:51.961966 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60cb912d-fn7bk"] Oct 30 22:04:51.962731 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:51.962707 5748 cpu_manager.go:414] "containerMap: removing container" podUID="b659bb43-102b-4936-87ed-d1b955262349" containerName="istio-proxy" Oct 30 22:04:51.962731 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:51.962728 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b659bb43-102b-4936-87ed-d1b955262349" containerName="istio-proxy" Oct 30 22:04:51.962995 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:51.962778 5748 cpu_manager.go:414] "containerMap: removing container" podUID="d2cdb6d5-0628-435d-ade1-e0aa84440718" containerName="istio-proxy" Oct 30 22:04:51.962995 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:51.962785 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2cdb6d5-0628-435d-ade1-e0aa84440718" containerName="istio-proxy" Oct 30 22:04:51.962995 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:51.962796 5748 cpu_manager.go:414] "containerMap: removing container" podUID="b6c9d2bc-60d6-494c-af57-574e85c4861c" containerName="istio-proxy" Oct 30 22:04:51.962995 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:51.962801 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6c9d2bc-60d6-494c-af57-574e85c4861c" containerName="istio-proxy" Oct 30 22:04:51.962995 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:51.962930 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2cdb6d5-0628-435d-ade1-e0aa84440718" containerName="istio-proxy" Oct 30 22:04:51.962995 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:51.962944 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6c9d2bc-60d6-494c-af57-574e85c4861c" containerName="istio-proxy" Oct 30 22:04:51.962995 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:51.962961 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="b659bb43-102b-4936-87ed-d1b955262349" containerName="istio-proxy" Oct 30 22:04:51.965106 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:51.965089 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60cb912d-fn7bk" Oct 30 22:04:52.022600 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:52.022537 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:04:52.022600 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:52.022604 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:04:52.022847 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:52.022698 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:04:52.023261 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:52.023235 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 22:04:52.026021 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:52.026001 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60cb912d-fn7bk"] Oct 30 22:04:52.116037 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:52.116000 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/994860af-7926-4f82-bd6d-616421b46611-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60cb912d-fn7bk\" (UID: \"994860af-7926-4f82-bd6d-616421b46611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60cb912d-fn7bk" Oct 30 22:04:52.116184 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:52.116112 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf56j\" (UniqueName: \"kubernetes.io/projected/994860af-7926-4f82-bd6d-616421b46611-kube-api-access-gf56j\") pod \"maas-api-revoke-sub-60cb912d-fn7bk\" (UID: \"994860af-7926-4f82-bd6d-616421b46611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60cb912d-fn7bk" Oct 30 22:04:52.116235 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:52.116186 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/994860af-7926-4f82-bd6d-616421b46611-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60cb912d-fn7bk\" (UID: \"994860af-7926-4f82-bd6d-616421b46611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60cb912d-fn7bk" Oct 30 22:04:52.217251 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:52.217177 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/994860af-7926-4f82-bd6d-616421b46611-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60cb912d-fn7bk\" (UID: \"994860af-7926-4f82-bd6d-616421b46611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60cb912d-fn7bk" Oct 30 22:04:52.217251 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:52.217246 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf56j\" (UniqueName: \"kubernetes.io/projected/994860af-7926-4f82-bd6d-616421b46611-kube-api-access-gf56j\") pod \"maas-api-revoke-sub-60cb912d-fn7bk\" (UID: \"994860af-7926-4f82-bd6d-616421b46611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60cb912d-fn7bk" Oct 30 22:04:52.217425 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:52.217295 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/994860af-7926-4f82-bd6d-616421b46611-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60cb912d-fn7bk\" (UID: \"994860af-7926-4f82-bd6d-616421b46611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60cb912d-fn7bk" Oct 30 22:04:52.217952 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:52.217930 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/994860af-7926-4f82-bd6d-616421b46611-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60cb912d-fn7bk\" (UID: \"994860af-7926-4f82-bd6d-616421b46611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60cb912d-fn7bk" Oct 30 22:04:52.219624 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:52.219600 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/994860af-7926-4f82-bd6d-616421b46611-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60cb912d-fn7bk\" (UID: \"994860af-7926-4f82-bd6d-616421b46611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60cb912d-fn7bk" Oct 30 22:04:52.279123 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:52.279099 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf56j\" (UniqueName: \"kubernetes.io/projected/994860af-7926-4f82-bd6d-616421b46611-kube-api-access-gf56j\") pod \"maas-api-revoke-sub-60cb912d-fn7bk\" (UID: \"994860af-7926-4f82-bd6d-616421b46611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60cb912d-fn7bk" Oct 30 22:04:52.578108 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:52.578024 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60cb912d-fn7bk" Oct 30 22:04:52.588248 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:52.588213 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v" event={"ID":"024f007a-0f46-4093-8554-93a335039f10","Type":"ContainerStarted","Data":"a07053a9af92e329f37abf16ea55d304cf51778fe8301ed34cedca2613fe3319"} Oct 30 22:04:52.681174 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:52.681148 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v" Oct 30 22:04:52.682208 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:52.682182 5748 patch_prober.go:29] interesting pod/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 30 22:04:52.682266 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:52.682242 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v" podUID="024f007a-0f46-4093-8554-93a335039f10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 30 22:04:52.986777 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:52.986749 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60cb912d-fn7bk"] Oct 30 22:04:52.988735 ip-10-0-143-61 kubenswrapper[5748]: W1030 22:04:52.988700 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod994860af_7926_4f82_bd6d_616421b46611.slice/crio-246c9a33a7cb7582667f7cd6080df723348842d334058bdd6bbb4cc3c56ebbd1 WatchSource:0}: Error finding container 246c9a33a7cb7582667f7cd6080df723348842d334058bdd6bbb4cc3c56ebbd1: Status 404 returned error can't find the container with id 246c9a33a7cb7582667f7cd6080df723348842d334058bdd6bbb4cc3c56ebbd1 Oct 30 22:04:53.595840 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:53.595804 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60cb912d-fn7bk" event={"ID":"994860af-7926-4f82-bd6d-616421b46611","Type":"ContainerStarted","Data":"c88f3c0140646a4fbaa46b19401b696c9737cbf9f08a30a377df9378ba23d284"} Oct 30 22:04:53.596298 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:53.595864 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60cb912d-fn7bk" event={"ID":"994860af-7926-4f82-bd6d-616421b46611","Type":"ContainerStarted","Data":"246c9a33a7cb7582667f7cd6080df723348842d334058bdd6bbb4cc3c56ebbd1"} Oct 30 22:04:53.673484 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:53.673449 5748 patch_prober.go:29] interesting pod/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 30 22:04:53.673654 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:53.673498 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v" podUID="024f007a-0f46-4093-8554-93a335039f10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 30 22:04:53.726980 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:53.724031 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60cb912d-fn7bk" podStartSLOduration=2.7240125060000002 podStartE2EDuration="2.724012506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:53.722992456 +0000 UTC m=+2477.465513393" watchObservedRunningTime="2026-10-30 22:04:53.724012506 +0000 UTC m=+2477.466533442" Oct 30 22:04:54.632007 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:54.631910 5748 generic.go:356] "Generic (PLEG): container finished" podID="994860af-7926-4f82-bd6d-616421b46611" containerID="c88f3c0140646a4fbaa46b19401b696c9737cbf9f08a30a377df9378ba23d284" exitCode=0 Oct 30 22:04:54.632586 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:54.632556 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60cb912d-fn7bk" event={"ID":"994860af-7926-4f82-bd6d-616421b46611","Type":"ContainerDied","Data":"c88f3c0140646a4fbaa46b19401b696c9737cbf9f08a30a377df9378ba23d284"} Oct 30 22:04:54.673458 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:54.673430 5748 patch_prober.go:29] interesting pod/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 30 22:04:54.673588 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:54.673479 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v" podUID="024f007a-0f46-4093-8554-93a335039f10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 30 22:04:54.678594 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:54.678574 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v" Oct 30 22:04:55.673037 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:55.672997 5748 patch_prober.go:29] interesting pod/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 30 22:04:55.673404 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:55.673054 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v" podUID="024f007a-0f46-4093-8554-93a335039f10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 30 22:04:55.829550 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:55.829523 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60cb912d-fn7bk" Oct 30 22:04:55.864262 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:55.864233 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/994860af-7926-4f82-bd6d-616421b46611-kube-api-access-gf56j\" (UniqueName: \"kubernetes.io/projected/994860af-7926-4f82-bd6d-616421b46611-kube-api-access-gf56j\") pod \"994860af-7926-4f82-bd6d-616421b46611\" (UID: \"994860af-7926-4f82-bd6d-616421b46611\") " Oct 30 22:04:55.864435 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:55.864368 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/994860af-7926-4f82-bd6d-616421b46611-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/994860af-7926-4f82-bd6d-616421b46611-maas-api-xks-ca\") pod \"994860af-7926-4f82-bd6d-616421b46611\" (UID: \"994860af-7926-4f82-bd6d-616421b46611\") " Oct 30 22:04:55.864562 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:55.864548 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/994860af-7926-4f82-bd6d-616421b46611-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/994860af-7926-4f82-bd6d-616421b46611-maas-api-openshift-ca\") pod \"994860af-7926-4f82-bd6d-616421b46611\" (UID: \"994860af-7926-4f82-bd6d-616421b46611\") " Oct 30 22:04:55.864881 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:55.864845 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/994860af-7926-4f82-bd6d-616421b46611-maas-api-openshift-ca" pod "994860af-7926-4f82-bd6d-616421b46611" (UID: "994860af-7926-4f82-bd6d-616421b46611"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:55.866340 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:55.866322 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/994860af-7926-4f82-bd6d-616421b46611-maas-api-xks-ca" pod "994860af-7926-4f82-bd6d-616421b46611" (UID: "994860af-7926-4f82-bd6d-616421b46611"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:04:55.866444 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:55.866378 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/994860af-7926-4f82-bd6d-616421b46611-kube-api-access-gf56j" pod "994860af-7926-4f82-bd6d-616421b46611" (UID: "994860af-7926-4f82-bd6d-616421b46611"). InnerVolumeSpecName "kube-api-access-gf56j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:55.965659 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:55.965588 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/994860af-7926-4f82-bd6d-616421b46611-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:55.965659 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:55.965616 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf56j\" (UniqueName: \"kubernetes.io/projected/994860af-7926-4f82-bd6d-616421b46611-kube-api-access-gf56j\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:55.965659 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:55.965628 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/994860af-7926-4f82-bd6d-616421b46611-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:56.644911 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:56.644864 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60cb912d-fn7bk" Oct 30 22:04:56.645067 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:56.644930 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60cb912d-fn7bk" event={"ID":"994860af-7926-4f82-bd6d-616421b46611","Type":"ContainerDied","Data":"246c9a33a7cb7582667f7cd6080df723348842d334058bdd6bbb4cc3c56ebbd1"} Oct 30 22:04:56.645067 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:56.644982 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="246c9a33a7cb7582667f7cd6080df723348842d334058bdd6bbb4cc3c56ebbd1" Oct 30 22:04:56.672970 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:56.672943 5748 patch_prober.go:29] interesting pod/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 30 22:04:56.673123 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:56.673000 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v" podUID="024f007a-0f46-4093-8554-93a335039f10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 30 22:04:57.677993 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:57.677013 5748 patch_prober.go:29] interesting pod/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 30 22:04:57.677993 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:57.677083 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v" podUID="024f007a-0f46-4093-8554-93a335039f10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 30 22:04:58.678411 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:58.678374 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v" Oct 30 22:04:58.679569 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:58.679546 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v" Oct 30 22:04:58.936236 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:58.936116 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8c5350-fc5mw"] Oct 30 22:04:58.936712 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:58.936695 5748 cpu_manager.go:414] "containerMap: removing container" podUID="994860af-7926-4f82-bd6d-616421b46611" containerName="revoke-keys" Oct 30 22:04:58.936767 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:58.936713 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="994860af-7926-4f82-bd6d-616421b46611" containerName="revoke-keys" Oct 30 22:04:58.936809 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:58.936796 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="994860af-7926-4f82-bd6d-616421b46611" containerName="revoke-keys" Oct 30 22:04:58.940090 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:58.940074 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8c5350-fc5mw" Oct 30 22:04:58.997082 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:58.997045 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:04:58.997474 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:58.997412 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:04:58.999204 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:58.999182 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 22:04:58.999354 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:58.999198 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:04:59.000492 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:59.000464 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8c5350-fc5mw"] Oct 30 22:04:59.005195 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:59.005152 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84255ff8-1706-49bc-aefa-f2c74bdfe0f1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8c5350-fc5mw\" (UID: \"84255ff8-1706-49bc-aefa-f2c74bdfe0f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8c5350-fc5mw" Oct 30 22:04:59.005301 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:59.005243 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84255ff8-1706-49bc-aefa-f2c74bdfe0f1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8c5350-fc5mw\" (UID: \"84255ff8-1706-49bc-aefa-f2c74bdfe0f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8c5350-fc5mw" Oct 30 22:04:59.005388 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:59.005373 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grbjh\" (UniqueName: \"kubernetes.io/projected/84255ff8-1706-49bc-aefa-f2c74bdfe0f1-kube-api-access-grbjh\") pod \"maas-api-revoke-keys-e2e-inf-a-8c5350-fc5mw\" (UID: \"84255ff8-1706-49bc-aefa-f2c74bdfe0f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8c5350-fc5mw" Oct 30 22:04:59.106963 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:59.106926 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84255ff8-1706-49bc-aefa-f2c74bdfe0f1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8c5350-fc5mw\" (UID: \"84255ff8-1706-49bc-aefa-f2c74bdfe0f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8c5350-fc5mw" Oct 30 22:04:59.107133 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:59.106990 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84255ff8-1706-49bc-aefa-f2c74bdfe0f1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8c5350-fc5mw\" (UID: \"84255ff8-1706-49bc-aefa-f2c74bdfe0f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8c5350-fc5mw" Oct 30 22:04:59.107133 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:59.107052 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grbjh\" (UniqueName: \"kubernetes.io/projected/84255ff8-1706-49bc-aefa-f2c74bdfe0f1-kube-api-access-grbjh\") pod \"maas-api-revoke-keys-e2e-inf-a-8c5350-fc5mw\" (UID: \"84255ff8-1706-49bc-aefa-f2c74bdfe0f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8c5350-fc5mw" Oct 30 22:04:59.107541 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:59.107517 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84255ff8-1706-49bc-aefa-f2c74bdfe0f1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8c5350-fc5mw\" (UID: \"84255ff8-1706-49bc-aefa-f2c74bdfe0f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8c5350-fc5mw" Oct 30 22:04:59.109285 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:59.109260 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84255ff8-1706-49bc-aefa-f2c74bdfe0f1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8c5350-fc5mw\" (UID: \"84255ff8-1706-49bc-aefa-f2c74bdfe0f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8c5350-fc5mw" Oct 30 22:04:59.168735 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:59.168707 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grbjh\" (UniqueName: \"kubernetes.io/projected/84255ff8-1706-49bc-aefa-f2c74bdfe0f1-kube-api-access-grbjh\") pod \"maas-api-revoke-keys-e2e-inf-a-8c5350-fc5mw\" (UID: \"84255ff8-1706-49bc-aefa-f2c74bdfe0f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8c5350-fc5mw" Oct 30 22:04:59.254003 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:59.253933 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8c5350-fc5mw" Oct 30 22:04:59.630195 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:59.630168 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8c5350-fc5mw"] Oct 30 22:04:59.632515 ip-10-0-143-61 kubenswrapper[5748]: W1030 22:04:59.632489 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84255ff8_1706_49bc_aefa_f2c74bdfe0f1.slice/crio-9e7fca7e8aacb52afbb2d7d7f28f3385a77a4d4cc711ed9485b30555832cc6b1 WatchSource:0}: Error finding container 9e7fca7e8aacb52afbb2d7d7f28f3385a77a4d4cc711ed9485b30555832cc6b1: Status 404 returned error can't find the container with id 9e7fca7e8aacb52afbb2d7d7f28f3385a77a4d4cc711ed9485b30555832cc6b1 Oct 30 22:04:59.666014 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:04:59.665989 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8c5350-fc5mw" event={"ID":"84255ff8-1706-49bc-aefa-f2c74bdfe0f1","Type":"ContainerStarted","Data":"9e7fca7e8aacb52afbb2d7d7f28f3385a77a4d4cc711ed9485b30555832cc6b1"} Oct 30 22:05:00.145812 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:05:00.145773 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:05:00.673558 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:00.673527 5748 generic.go:356] "Generic (PLEG): container finished" podID="84255ff8-1706-49bc-aefa-f2c74bdfe0f1" containerID="5243e6663aee3382320cf32fff162004e5c2042e278384c08b1e333f804c16c2" exitCode=0 Oct 30 22:05:00.673705 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:00.673610 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8c5350-fc5mw" event={"ID":"84255ff8-1706-49bc-aefa-f2c74bdfe0f1","Type":"ContainerDied","Data":"5243e6663aee3382320cf32fff162004e5c2042e278384c08b1e333f804c16c2"} Oct 30 22:05:01.844467 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:01.844441 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8c5350-fc5mw" Oct 30 22:05:01.940391 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:01.940303 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84255ff8-1706-49bc-aefa-f2c74bdfe0f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84255ff8-1706-49bc-aefa-f2c74bdfe0f1-maas-api-xks-ca\") pod \"84255ff8-1706-49bc-aefa-f2c74bdfe0f1\" (UID: \"84255ff8-1706-49bc-aefa-f2c74bdfe0f1\") " Oct 30 22:05:01.940568 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:01.940426 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84255ff8-1706-49bc-aefa-f2c74bdfe0f1-kube-api-access-grbjh\" (UniqueName: \"kubernetes.io/projected/84255ff8-1706-49bc-aefa-f2c74bdfe0f1-kube-api-access-grbjh\") pod \"84255ff8-1706-49bc-aefa-f2c74bdfe0f1\" (UID: \"84255ff8-1706-49bc-aefa-f2c74bdfe0f1\") " Oct 30 22:05:01.940568 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:01.940484 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84255ff8-1706-49bc-aefa-f2c74bdfe0f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84255ff8-1706-49bc-aefa-f2c74bdfe0f1-maas-api-openshift-ca\") pod \"84255ff8-1706-49bc-aefa-f2c74bdfe0f1\" (UID: \"84255ff8-1706-49bc-aefa-f2c74bdfe0f1\") " Oct 30 22:05:01.940874 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:01.940845 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84255ff8-1706-49bc-aefa-f2c74bdfe0f1-maas-api-openshift-ca" pod "84255ff8-1706-49bc-aefa-f2c74bdfe0f1" (UID: "84255ff8-1706-49bc-aefa-f2c74bdfe0f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:05:01.941089 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:01.941065 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84255ff8-1706-49bc-aefa-f2c74bdfe0f1-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:01.942541 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:01.942513 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84255ff8-1706-49bc-aefa-f2c74bdfe0f1-kube-api-access-grbjh" pod "84255ff8-1706-49bc-aefa-f2c74bdfe0f1" (UID: "84255ff8-1706-49bc-aefa-f2c74bdfe0f1"). InnerVolumeSpecName "kube-api-access-grbjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:05:01.942636 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:01.942613 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84255ff8-1706-49bc-aefa-f2c74bdfe0f1-maas-api-xks-ca" pod "84255ff8-1706-49bc-aefa-f2c74bdfe0f1" (UID: "84255ff8-1706-49bc-aefa-f2c74bdfe0f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:05:02.042990 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:02.042959 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84255ff8-1706-49bc-aefa-f2c74bdfe0f1-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:02.042990 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:02.042985 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grbjh\" (UniqueName: \"kubernetes.io/projected/84255ff8-1706-49bc-aefa-f2c74bdfe0f1-kube-api-access-grbjh\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:02.690008 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:02.689923 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8c5350-fc5mw" event={"ID":"84255ff8-1706-49bc-aefa-f2c74bdfe0f1","Type":"ContainerDied","Data":"9e7fca7e8aacb52afbb2d7d7f28f3385a77a4d4cc711ed9485b30555832cc6b1"} Oct 30 22:05:02.690008 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:02.690002 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e7fca7e8aacb52afbb2d7d7f28f3385a77a4d4cc711ed9485b30555832cc6b1" Oct 30 22:05:02.690195 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:02.689935 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8c5350-fc5mw" Oct 30 22:05:03.317540 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:03.317503 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d68156-gqhpp"] Oct 30 22:05:03.318082 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:03.318066 5748 cpu_manager.go:414] "containerMap: removing container" podUID="84255ff8-1706-49bc-aefa-f2c74bdfe0f1" containerName="revoke-keys" Oct 30 22:05:03.318126 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:03.318084 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84255ff8-1706-49bc-aefa-f2c74bdfe0f1" containerName="revoke-keys" Oct 30 22:05:03.318209 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:03.318200 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="84255ff8-1706-49bc-aefa-f2c74bdfe0f1" containerName="revoke-keys" Oct 30 22:05:03.321775 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:03.321749 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d68156-gqhpp" Oct 30 22:05:03.378072 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:03.378028 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:05:03.378262 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:03.378162 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:05:03.378262 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:03.378212 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:05:03.378397 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:03.378282 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 22:05:03.380647 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:03.380621 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d68156-gqhpp"] Oct 30 22:05:03.460890 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:03.460858 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59df8c0f-237e-4d38-9d2d-0982d5f44743-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-d68156-gqhpp\" (UID: \"59df8c0f-237e-4d38-9d2d-0982d5f44743\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d68156-gqhpp" Oct 30 22:05:03.461176 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:03.461141 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59df8c0f-237e-4d38-9d2d-0982d5f44743-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-d68156-gqhpp\" (UID: \"59df8c0f-237e-4d38-9d2d-0982d5f44743\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d68156-gqhpp" Oct 30 22:05:03.461313 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:03.461196 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb87m\" (UniqueName: \"kubernetes.io/projected/59df8c0f-237e-4d38-9d2d-0982d5f44743-kube-api-access-mb87m\") pod \"maas-api-revoke-keys-e2e-worker-w3-d68156-gqhpp\" (UID: \"59df8c0f-237e-4d38-9d2d-0982d5f44743\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d68156-gqhpp" Oct 30 22:05:03.562042 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:03.562002 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59df8c0f-237e-4d38-9d2d-0982d5f44743-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-d68156-gqhpp\" (UID: \"59df8c0f-237e-4d38-9d2d-0982d5f44743\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d68156-gqhpp" Oct 30 22:05:03.562199 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:03.562146 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59df8c0f-237e-4d38-9d2d-0982d5f44743-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-d68156-gqhpp\" (UID: \"59df8c0f-237e-4d38-9d2d-0982d5f44743\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d68156-gqhpp" Oct 30 22:05:03.562199 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:03.562174 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb87m\" (UniqueName: \"kubernetes.io/projected/59df8c0f-237e-4d38-9d2d-0982d5f44743-kube-api-access-mb87m\") pod \"maas-api-revoke-keys-e2e-worker-w3-d68156-gqhpp\" (UID: \"59df8c0f-237e-4d38-9d2d-0982d5f44743\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d68156-gqhpp" Oct 30 22:05:03.562814 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:03.562787 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59df8c0f-237e-4d38-9d2d-0982d5f44743-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-d68156-gqhpp\" (UID: \"59df8c0f-237e-4d38-9d2d-0982d5f44743\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d68156-gqhpp" Oct 30 22:05:03.564494 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:03.564468 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59df8c0f-237e-4d38-9d2d-0982d5f44743-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-d68156-gqhpp\" (UID: \"59df8c0f-237e-4d38-9d2d-0982d5f44743\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d68156-gqhpp" Oct 30 22:05:03.623314 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:03.623260 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb87m\" (UniqueName: \"kubernetes.io/projected/59df8c0f-237e-4d38-9d2d-0982d5f44743-kube-api-access-mb87m\") pod \"maas-api-revoke-keys-e2e-worker-w3-d68156-gqhpp\" (UID: \"59df8c0f-237e-4d38-9d2d-0982d5f44743\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d68156-gqhpp" Oct 30 22:05:03.635032 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:03.635012 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d68156-gqhpp" Oct 30 22:05:03.941258 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:03.941226 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-ea1f0b/test-model-ea1f0b-kserve-586698c74f-dmv5g"] Oct 30 22:05:03.946394 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:03.946376 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ea1f0b/test-model-ea1f0b-kserve-586698c74f-dmv5g" Oct 30 22:05:04.006852 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:04.006817 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ea1f0b/test-model-ea1f0b-kserve-586698c74f-dmv5g"] Oct 30 22:05:04.012025 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:04.011998 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ea1f0b\"/\"kube-root-ca.crt\"" Oct 30 22:05:04.012025 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:04.012005 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ea1f0b\"/\"openshift-service-ca.crt\"" Oct 30 22:05:04.012271 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:04.012090 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ea1f0b\"/\"default-dockercfg-pt62n\"" Oct 30 22:05:04.012271 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:04.012188 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ea1f0b\"/\"test-model-ea1f0b-kserve-self-signed-certs\"" Oct 30 22:05:04.054800 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:04.054780 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d68156-gqhpp"] Oct 30 22:05:04.056351 ip-10-0-143-61 kubenswrapper[5748]: W1030 22:05:04.056326 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59df8c0f_237e_4d38_9d2d_0982d5f44743.slice/crio-6b3c60e9145e085cd3b3d4a364e589da9f6abf547b4d6a856e171c09568db84f WatchSource:0}: Error finding container 6b3c60e9145e085cd3b3d4a364e589da9f6abf547b4d6a856e171c09568db84f: Status 404 returned error can't find the container with id 6b3c60e9145e085cd3b3d4a364e589da9f6abf547b4d6a856e171c09568db84f Oct 30 22:05:04.067552 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:04.067526 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpkt8\" (UniqueName: \"kubernetes.io/projected/a8257bca-a09a-415f-9616-86bdc26b79b1-kube-api-access-gpkt8\") pod \"test-model-ea1f0b-kserve-586698c74f-dmv5g\" (UID: \"a8257bca-a09a-415f-9616-86bdc26b79b1\") " pod="ai-tenant-e2e-inf-b-ea1f0b/test-model-ea1f0b-kserve-586698c74f-dmv5g" Oct 30 22:05:04.067667 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:04.067574 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8257bca-a09a-415f-9616-86bdc26b79b1-tls-certs\") pod \"test-model-ea1f0b-kserve-586698c74f-dmv5g\" (UID: \"a8257bca-a09a-415f-9616-86bdc26b79b1\") " pod="ai-tenant-e2e-inf-b-ea1f0b/test-model-ea1f0b-kserve-586698c74f-dmv5g" Oct 30 22:05:04.067728 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:04.067684 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8257bca-a09a-415f-9616-86bdc26b79b1-model-cache\") pod \"test-model-ea1f0b-kserve-586698c74f-dmv5g\" (UID: \"a8257bca-a09a-415f-9616-86bdc26b79b1\") " pod="ai-tenant-e2e-inf-b-ea1f0b/test-model-ea1f0b-kserve-586698c74f-dmv5g" Oct 30 22:05:04.067792 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:04.067724 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8257bca-a09a-415f-9616-86bdc26b79b1-dshm\") pod \"test-model-ea1f0b-kserve-586698c74f-dmv5g\" (UID: \"a8257bca-a09a-415f-9616-86bdc26b79b1\") " pod="ai-tenant-e2e-inf-b-ea1f0b/test-model-ea1f0b-kserve-586698c74f-dmv5g" Oct 30 22:05:04.067792 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:04.067754 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8257bca-a09a-415f-9616-86bdc26b79b1-tmp-dir\") pod \"test-model-ea1f0b-kserve-586698c74f-dmv5g\" (UID: \"a8257bca-a09a-415f-9616-86bdc26b79b1\") " pod="ai-tenant-e2e-inf-b-ea1f0b/test-model-ea1f0b-kserve-586698c74f-dmv5g" Oct 30 22:05:04.067913 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:04.067833 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8257bca-a09a-415f-9616-86bdc26b79b1-home\") pod \"test-model-ea1f0b-kserve-586698c74f-dmv5g\" (UID: \"a8257bca-a09a-415f-9616-86bdc26b79b1\") " pod="ai-tenant-e2e-inf-b-ea1f0b/test-model-ea1f0b-kserve-586698c74f-dmv5g" Oct 30 22:05:04.169171 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:04.169138 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8257bca-a09a-415f-9616-86bdc26b79b1-tls-certs\") pod \"test-model-ea1f0b-kserve-586698c74f-dmv5g\" (UID: \"a8257bca-a09a-415f-9616-86bdc26b79b1\") " pod="ai-tenant-e2e-inf-b-ea1f0b/test-model-ea1f0b-kserve-586698c74f-dmv5g" Oct 30 22:05:04.169390 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:04.169238 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8257bca-a09a-415f-9616-86bdc26b79b1-model-cache\") pod \"test-model-ea1f0b-kserve-586698c74f-dmv5g\" (UID: \"a8257bca-a09a-415f-9616-86bdc26b79b1\") " pod="ai-tenant-e2e-inf-b-ea1f0b/test-model-ea1f0b-kserve-586698c74f-dmv5g" Oct 30 22:05:04.169390 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:04.169283 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8257bca-a09a-415f-9616-86bdc26b79b1-dshm\") pod \"test-model-ea1f0b-kserve-586698c74f-dmv5g\" (UID: \"a8257bca-a09a-415f-9616-86bdc26b79b1\") " pod="ai-tenant-e2e-inf-b-ea1f0b/test-model-ea1f0b-kserve-586698c74f-dmv5g" Oct 30 22:05:04.169390 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:04.169321 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8257bca-a09a-415f-9616-86bdc26b79b1-tmp-dir\") pod \"test-model-ea1f0b-kserve-586698c74f-dmv5g\" (UID: \"a8257bca-a09a-415f-9616-86bdc26b79b1\") " pod="ai-tenant-e2e-inf-b-ea1f0b/test-model-ea1f0b-kserve-586698c74f-dmv5g" Oct 30 22:05:04.169390 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:04.169352 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8257bca-a09a-415f-9616-86bdc26b79b1-home\") pod \"test-model-ea1f0b-kserve-586698c74f-dmv5g\" (UID: \"a8257bca-a09a-415f-9616-86bdc26b79b1\") " pod="ai-tenant-e2e-inf-b-ea1f0b/test-model-ea1f0b-kserve-586698c74f-dmv5g" Oct 30 22:05:04.169625 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:04.169397 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpkt8\" (UniqueName: \"kubernetes.io/projected/a8257bca-a09a-415f-9616-86bdc26b79b1-kube-api-access-gpkt8\") pod \"test-model-ea1f0b-kserve-586698c74f-dmv5g\" (UID: \"a8257bca-a09a-415f-9616-86bdc26b79b1\") " pod="ai-tenant-e2e-inf-b-ea1f0b/test-model-ea1f0b-kserve-586698c74f-dmv5g" Oct 30 22:05:04.169771 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:04.169747 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8257bca-a09a-415f-9616-86bdc26b79b1-model-cache\") pod \"test-model-ea1f0b-kserve-586698c74f-dmv5g\" (UID: \"a8257bca-a09a-415f-9616-86bdc26b79b1\") " pod="ai-tenant-e2e-inf-b-ea1f0b/test-model-ea1f0b-kserve-586698c74f-dmv5g" Oct 30 22:05:04.169831 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:04.169769 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8257bca-a09a-415f-9616-86bdc26b79b1-home\") pod \"test-model-ea1f0b-kserve-586698c74f-dmv5g\" (UID: \"a8257bca-a09a-415f-9616-86bdc26b79b1\") " pod="ai-tenant-e2e-inf-b-ea1f0b/test-model-ea1f0b-kserve-586698c74f-dmv5g" Oct 30 22:05:04.169909 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:04.169823 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8257bca-a09a-415f-9616-86bdc26b79b1-tmp-dir\") pod \"test-model-ea1f0b-kserve-586698c74f-dmv5g\" (UID: \"a8257bca-a09a-415f-9616-86bdc26b79b1\") " pod="ai-tenant-e2e-inf-b-ea1f0b/test-model-ea1f0b-kserve-586698c74f-dmv5g" Oct 30 22:05:04.171563 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:04.171539 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8257bca-a09a-415f-9616-86bdc26b79b1-dshm\") pod \"test-model-ea1f0b-kserve-586698c74f-dmv5g\" (UID: \"a8257bca-a09a-415f-9616-86bdc26b79b1\") " pod="ai-tenant-e2e-inf-b-ea1f0b/test-model-ea1f0b-kserve-586698c74f-dmv5g" Oct 30 22:05:04.171714 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:04.171696 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8257bca-a09a-415f-9616-86bdc26b79b1-tls-certs\") pod \"test-model-ea1f0b-kserve-586698c74f-dmv5g\" (UID: \"a8257bca-a09a-415f-9616-86bdc26b79b1\") " pod="ai-tenant-e2e-inf-b-ea1f0b/test-model-ea1f0b-kserve-586698c74f-dmv5g" Oct 30 22:05:04.236826 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:04.236759 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpkt8\" (UniqueName: \"kubernetes.io/projected/a8257bca-a09a-415f-9616-86bdc26b79b1-kube-api-access-gpkt8\") pod \"test-model-ea1f0b-kserve-586698c74f-dmv5g\" (UID: \"a8257bca-a09a-415f-9616-86bdc26b79b1\") " pod="ai-tenant-e2e-inf-b-ea1f0b/test-model-ea1f0b-kserve-586698c74f-dmv5g" Oct 30 22:05:04.259940 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:04.259912 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ea1f0b/test-model-ea1f0b-kserve-586698c74f-dmv5g" Oct 30 22:05:04.648567 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:04.648537 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ea1f0b/test-model-ea1f0b-kserve-586698c74f-dmv5g"] Oct 30 22:05:04.650503 ip-10-0-143-61 kubenswrapper[5748]: W1030 22:05:04.650474 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda8257bca_a09a_415f_9616_86bdc26b79b1.slice/crio-ff2aeda6e7a878ec075a00b7e9cc0604365556953991a7006c44335c95a77f46 WatchSource:0}: Error finding container ff2aeda6e7a878ec075a00b7e9cc0604365556953991a7006c44335c95a77f46: Status 404 returned error can't find the container with id ff2aeda6e7a878ec075a00b7e9cc0604365556953991a7006c44335c95a77f46 Oct 30 22:05:04.708625 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:04.708591 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d68156-gqhpp" event={"ID":"59df8c0f-237e-4d38-9d2d-0982d5f44743","Type":"ContainerStarted","Data":"52f65eb7588132cfe7b918958c0ef17ef09f9344fa0d54d4f2a8088700350dc8"} Oct 30 22:05:04.708760 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:04.708655 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d68156-gqhpp" event={"ID":"59df8c0f-237e-4d38-9d2d-0982d5f44743","Type":"ContainerStarted","Data":"6b3c60e9145e085cd3b3d4a364e589da9f6abf547b4d6a856e171c09568db84f"} Oct 30 22:05:04.709918 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:04.709869 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ea1f0b/test-model-ea1f0b-kserve-586698c74f-dmv5g" event={"ID":"a8257bca-a09a-415f-9616-86bdc26b79b1","Type":"ContainerStarted","Data":"ff2aeda6e7a878ec075a00b7e9cc0604365556953991a7006c44335c95a77f46"} Oct 30 22:05:04.828586 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:04.828530 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d68156-gqhpp" podStartSLOduration=1.828516562 podStartE2EDuration="1.828516562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:04.825758357 +0000 UTC m=+2488.568279292" watchObservedRunningTime="2026-10-30 22:05:04.828516562 +0000 UTC m=+2488.571037497" Oct 30 22:05:05.717769 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:05.717732 5748 generic.go:356] "Generic (PLEG): container finished" podID="59df8c0f-237e-4d38-9d2d-0982d5f44743" containerID="52f65eb7588132cfe7b918958c0ef17ef09f9344fa0d54d4f2a8088700350dc8" exitCode=0 Oct 30 22:05:05.718218 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:05.717822 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d68156-gqhpp" event={"ID":"59df8c0f-237e-4d38-9d2d-0982d5f44743","Type":"ContainerDied","Data":"52f65eb7588132cfe7b918958c0ef17ef09f9344fa0d54d4f2a8088700350dc8"} Oct 30 22:05:05.719456 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:05.719426 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ea1f0b/test-model-ea1f0b-kserve-586698c74f-dmv5g" event={"ID":"a8257bca-a09a-415f-9616-86bdc26b79b1","Type":"ContainerStarted","Data":"14d286bb030531ec871ffbbb5dd48aa91b1fb5fdad5ffb8578762783cd0e5e34"} Oct 30 22:05:05.960118 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:05.960061 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-ea1f0b/test-model-ea1f0b-kserve-586698c74f-dmv5g" podStartSLOduration=2.960045025 podStartE2EDuration="2.960045025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:05.958241311 +0000 UTC m=+2489.700762257" watchObservedRunningTime="2026-10-30 22:05:05.960045025 +0000 UTC m=+2489.702565959" Oct 30 22:05:06.260413 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:06.260378 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-ea1f0b/test-model-ea1f0b-kserve-586698c74f-dmv5g" Oct 30 22:05:06.887354 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:06.887327 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d68156-gqhpp" Oct 30 22:05:07.022132 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:07.022045 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59df8c0f-237e-4d38-9d2d-0982d5f44743-kube-api-access-mb87m\" (UniqueName: \"kubernetes.io/projected/59df8c0f-237e-4d38-9d2d-0982d5f44743-kube-api-access-mb87m\") pod \"59df8c0f-237e-4d38-9d2d-0982d5f44743\" (UID: \"59df8c0f-237e-4d38-9d2d-0982d5f44743\") " Oct 30 22:05:07.022132 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:07.022086 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59df8c0f-237e-4d38-9d2d-0982d5f44743-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59df8c0f-237e-4d38-9d2d-0982d5f44743-maas-api-openshift-ca\") pod \"59df8c0f-237e-4d38-9d2d-0982d5f44743\" (UID: \"59df8c0f-237e-4d38-9d2d-0982d5f44743\") " Oct 30 22:05:07.022132 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:07.022126 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59df8c0f-237e-4d38-9d2d-0982d5f44743-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59df8c0f-237e-4d38-9d2d-0982d5f44743-maas-api-xks-ca\") pod \"59df8c0f-237e-4d38-9d2d-0982d5f44743\" (UID: \"59df8c0f-237e-4d38-9d2d-0982d5f44743\") " Oct 30 22:05:07.022559 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:07.022532 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59df8c0f-237e-4d38-9d2d-0982d5f44743-maas-api-openshift-ca" pod "59df8c0f-237e-4d38-9d2d-0982d5f44743" (UID: "59df8c0f-237e-4d38-9d2d-0982d5f44743"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:05:07.022722 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:07.022695 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59df8c0f-237e-4d38-9d2d-0982d5f44743-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:07.024352 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:07.024327 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59df8c0f-237e-4d38-9d2d-0982d5f44743-maas-api-xks-ca" pod "59df8c0f-237e-4d38-9d2d-0982d5f44743" (UID: "59df8c0f-237e-4d38-9d2d-0982d5f44743"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:05:07.024588 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:07.024560 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59df8c0f-237e-4d38-9d2d-0982d5f44743-kube-api-access-mb87m" pod "59df8c0f-237e-4d38-9d2d-0982d5f44743" (UID: "59df8c0f-237e-4d38-9d2d-0982d5f44743"). InnerVolumeSpecName "kube-api-access-mb87m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:05:07.123787 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:07.123759 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59df8c0f-237e-4d38-9d2d-0982d5f44743-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:07.123787 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:07.123784 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb87m\" (UniqueName: \"kubernetes.io/projected/59df8c0f-237e-4d38-9d2d-0982d5f44743-kube-api-access-mb87m\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:07.695250 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:07.695219 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d68156/distinct-w3-d68156-kserve-565894fdb4-85t7z"] Oct 30 22:05:07.734363 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:07.734330 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d68156-gqhpp" Oct 30 22:05:07.734363 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:07.734342 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d68156-gqhpp" event={"ID":"59df8c0f-237e-4d38-9d2d-0982d5f44743","Type":"ContainerDied","Data":"6b3c60e9145e085cd3b3d4a364e589da9f6abf547b4d6a856e171c09568db84f"} Oct 30 22:05:07.734622 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:07.734400 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b3c60e9145e085cd3b3d4a364e589da9f6abf547b4d6a856e171c09568db84f" Oct 30 22:05:07.745814 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:07.745789 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d68156/simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk"] Oct 30 22:05:14.261112 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:14.261078 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-ea1f0b/test-model-ea1f0b-kserve-586698c74f-dmv5g" Oct 30 22:05:14.273254 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:14.273230 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-ea1f0b/test-model-ea1f0b-kserve-586698c74f-dmv5g" Oct 30 22:05:14.791672 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:14.791644 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-ea1f0b/test-model-ea1f0b-kserve-586698c74f-dmv5g" Oct 30 22:05:15.027668 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:15.027628 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-d68156-78b685b5dc-pwg7l"] Oct 30 22:05:20.028753 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:20.028704 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d68156-78b685b5dc-pwg7l" podUID="8b269e36-1397-47d9-a349-04304ad783a4" containerName="payload-pre-processing" containerID="cri-o://5560debf400a8ca000f0638c00abd098dd70e6616236a15d7727409a3fd3d78f" gracePeriod=25 Oct 30 22:05:22.717807 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:22.717760 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-d68156/distinct-w3-d68156-kserve-565894fdb4-85t7z" podUID="13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe" containerName="main" containerID="cri-o://486949e0219addc738f87a366f0ea90c243baa9a9c6cf4bf9ce86aed48906f0f" gracePeriod=45 Oct 30 22:05:22.722322 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:22.722283 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d68156/distinct-w3-d68156-kserve-565894fdb4-85t7z" podUID="13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": EOF" Oct 30 22:05:22.765284 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:22.765254 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-d68156/simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk" podUID="db285e9c-ca87-43fa-8f06-69e271e5c2c5" containerName="main" containerID="cri-o://648bf1c437fe90e4c4cbe008c5c76fc9c55248af538d84a3c0a526c84f5512e3" gracePeriod=45 Oct 30 22:05:22.811941 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:22.811879 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d68156/simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk" podUID="db285e9c-ca87-43fa-8f06-69e271e5c2c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": dial tcp 10.134.0.41:8000: connect: connection refused" Oct 30 22:05:22.834788 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:22.834762 5748 generic.go:356] "Generic (PLEG): container finished" podID="13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe" containerID="486949e0219addc738f87a366f0ea90c243baa9a9c6cf4bf9ce86aed48906f0f" exitCode=0 Oct 30 22:05:22.834937 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:22.834836 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d68156/distinct-w3-d68156-kserve-565894fdb4-85t7z" event={"ID":"13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe","Type":"ContainerDied","Data":"486949e0219addc738f87a366f0ea90c243baa9a9c6cf4bf9ce86aed48906f0f"} Oct 30 22:05:23.055249 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:23.054856 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d68156/distinct-w3-d68156-kserve-565894fdb4-85t7z" Oct 30 22:05:23.112439 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:23.112416 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d68156/simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk" Oct 30 22:05:23.119056 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:23.119032 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe-kube-api-access-q2bhb\" (UniqueName: \"kubernetes.io/projected/13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe-kube-api-access-q2bhb\") pod \"13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe\" (UID: \"13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe\") " Oct 30 22:05:23.119210 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:23.119137 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe-home\" (UniqueName: \"kubernetes.io/empty-dir/13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe-home\") pod \"13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe\" (UID: \"13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe\") " Oct 30 22:05:23.119210 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:23.119168 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe-tmp-dir\") pod \"13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe\" (UID: \"13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe\") " Oct 30 22:05:23.119210 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:23.119184 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe-dshm\" (UniqueName: \"kubernetes.io/empty-dir/13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe-dshm\") pod \"13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe\" (UID: \"13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe\") " Oct 30 22:05:23.119388 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:23.119286 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe-tls-certs\" (UniqueName: \"kubernetes.io/secret/13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe-tls-certs\") pod \"13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe\" (UID: \"13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe\") " Oct 30 22:05:23.120090 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:23.119446 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe-model-cache\") pod \"13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe\" (UID: \"13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe\") " Oct 30 22:05:23.120090 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:23.119483 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe-home" pod "13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe" (UID: "13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:23.120090 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:23.119655 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe-tmp-dir" pod "13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe" (UID: "13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:23.120090 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:23.119820 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe-model-cache" pod "13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe" (UID: "13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:23.120335 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:23.120242 5748 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe-home\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:23.120335 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:23.120264 5748 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe-tmp-dir\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:23.120335 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:23.120279 5748 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe-model-cache\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:23.122597 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:23.122543 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe-kube-api-access-q2bhb" pod "13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe" (UID: "13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe"). InnerVolumeSpecName "kube-api-access-q2bhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:05:23.123280 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:23.122731 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe-tls-certs" pod "13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe" (UID: "13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:05:23.123442 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:23.123417 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe-dshm" pod "13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe" (UID: "13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:23.220892 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:23.220853 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db285e9c-ca87-43fa-8f06-69e271e5c2c5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db285e9c-ca87-43fa-8f06-69e271e5c2c5-tmp-dir\") pod \"db285e9c-ca87-43fa-8f06-69e271e5c2c5\" (UID: \"db285e9c-ca87-43fa-8f06-69e271e5c2c5\") " Oct 30 22:05:23.221110 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:23.220967 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db285e9c-ca87-43fa-8f06-69e271e5c2c5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/db285e9c-ca87-43fa-8f06-69e271e5c2c5-dshm\") pod \"db285e9c-ca87-43fa-8f06-69e271e5c2c5\" (UID: \"db285e9c-ca87-43fa-8f06-69e271e5c2c5\") " Oct 30 22:05:23.221110 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:23.221058 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db285e9c-ca87-43fa-8f06-69e271e5c2c5-home\" (UniqueName: \"kubernetes.io/empty-dir/db285e9c-ca87-43fa-8f06-69e271e5c2c5-home\") pod \"db285e9c-ca87-43fa-8f06-69e271e5c2c5\" (UID: \"db285e9c-ca87-43fa-8f06-69e271e5c2c5\") " Oct 30 22:05:23.221236 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:23.221135 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db285e9c-ca87-43fa-8f06-69e271e5c2c5-tls-certs\" (UniqueName: \"kubernetes.io/secret/db285e9c-ca87-43fa-8f06-69e271e5c2c5-tls-certs\") pod \"db285e9c-ca87-43fa-8f06-69e271e5c2c5\" (UID: \"db285e9c-ca87-43fa-8f06-69e271e5c2c5\") " Oct 30 22:05:23.221236 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:23.221179 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db285e9c-ca87-43fa-8f06-69e271e5c2c5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db285e9c-ca87-43fa-8f06-69e271e5c2c5-model-cache\") pod \"db285e9c-ca87-43fa-8f06-69e271e5c2c5\" (UID: \"db285e9c-ca87-43fa-8f06-69e271e5c2c5\") " Oct 30 22:05:23.221357 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:23.221234 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db285e9c-ca87-43fa-8f06-69e271e5c2c5-kube-api-access-cvwll\" (UniqueName: \"kubernetes.io/projected/db285e9c-ca87-43fa-8f06-69e271e5c2c5-kube-api-access-cvwll\") pod \"db285e9c-ca87-43fa-8f06-69e271e5c2c5\" (UID: \"db285e9c-ca87-43fa-8f06-69e271e5c2c5\") " Oct 30 22:05:23.221771 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:23.221168 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db285e9c-ca87-43fa-8f06-69e271e5c2c5-tmp-dir" pod "db285e9c-ca87-43fa-8f06-69e271e5c2c5" (UID: "db285e9c-ca87-43fa-8f06-69e271e5c2c5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:23.221771 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:23.221367 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db285e9c-ca87-43fa-8f06-69e271e5c2c5-home" pod "db285e9c-ca87-43fa-8f06-69e271e5c2c5" (UID: "db285e9c-ca87-43fa-8f06-69e271e5c2c5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:23.221771 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:23.221540 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db285e9c-ca87-43fa-8f06-69e271e5c2c5-model-cache" pod "db285e9c-ca87-43fa-8f06-69e271e5c2c5" (UID: "db285e9c-ca87-43fa-8f06-69e271e5c2c5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:23.222087 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:23.221771 5748 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db285e9c-ca87-43fa-8f06-69e271e5c2c5-tmp-dir\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:23.222087 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:23.221793 5748 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db285e9c-ca87-43fa-8f06-69e271e5c2c5-home\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:23.222087 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:23.221808 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2bhb\" (UniqueName: \"kubernetes.io/projected/13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe-kube-api-access-q2bhb\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:23.222087 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:23.221823 5748 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db285e9c-ca87-43fa-8f06-69e271e5c2c5-model-cache\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:23.222087 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:23.221849 5748 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe-dshm\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:23.222087 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:23.221862 5748 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe-tls-certs\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:23.223669 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:23.223639 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db285e9c-ca87-43fa-8f06-69e271e5c2c5-dshm" pod "db285e9c-ca87-43fa-8f06-69e271e5c2c5" (UID: "db285e9c-ca87-43fa-8f06-69e271e5c2c5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:23.224234 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:23.224209 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db285e9c-ca87-43fa-8f06-69e271e5c2c5-kube-api-access-cvwll" pod "db285e9c-ca87-43fa-8f06-69e271e5c2c5" (UID: "db285e9c-ca87-43fa-8f06-69e271e5c2c5"). InnerVolumeSpecName "kube-api-access-cvwll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:05:23.224932 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:23.224539 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db285e9c-ca87-43fa-8f06-69e271e5c2c5-tls-certs" pod "db285e9c-ca87-43fa-8f06-69e271e5c2c5" (UID: "db285e9c-ca87-43fa-8f06-69e271e5c2c5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:05:23.323676 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:23.323586 5748 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db285e9c-ca87-43fa-8f06-69e271e5c2c5-dshm\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:23.323676 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:23.323627 5748 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db285e9c-ca87-43fa-8f06-69e271e5c2c5-tls-certs\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:23.323676 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:23.323642 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvwll\" (UniqueName: \"kubernetes.io/projected/db285e9c-ca87-43fa-8f06-69e271e5c2c5-kube-api-access-cvwll\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:23.842715 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:23.842679 5748 generic.go:356] "Generic (PLEG): container finished" podID="db285e9c-ca87-43fa-8f06-69e271e5c2c5" containerID="648bf1c437fe90e4c4cbe008c5c76fc9c55248af538d84a3c0a526c84f5512e3" exitCode=0 Oct 30 22:05:23.843177 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:23.842721 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d68156/simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk" Oct 30 22:05:23.843177 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:23.842761 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d68156/simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk" event={"ID":"db285e9c-ca87-43fa-8f06-69e271e5c2c5","Type":"ContainerDied","Data":"648bf1c437fe90e4c4cbe008c5c76fc9c55248af538d84a3c0a526c84f5512e3"} Oct 30 22:05:23.843177 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:23.842803 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d68156/simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk" event={"ID":"db285e9c-ca87-43fa-8f06-69e271e5c2c5","Type":"ContainerDied","Data":"cac752eb6b66afdf26b01a667ea0e6c22419b0a3eea717afe0ddf02a971c7cc0"} Oct 30 22:05:23.843177 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:23.842827 5748 scope.go:122] "RemoveContainer" containerID="648bf1c437fe90e4c4cbe008c5c76fc9c55248af538d84a3c0a526c84f5512e3" Oct 30 22:05:23.844578 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:23.844561 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d68156/distinct-w3-d68156-kserve-565894fdb4-85t7z" Oct 30 22:05:23.844714 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:23.844566 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d68156/distinct-w3-d68156-kserve-565894fdb4-85t7z" event={"ID":"13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe","Type":"ContainerDied","Data":"ad45feefa4e12d4aa7b883bd0eb899f9bb87b65a1842b1c349125bbe73137f0c"} Oct 30 22:05:23.853535 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:23.853470 5748 scope.go:122] "RemoveContainer" containerID="648bf1c437fe90e4c4cbe008c5c76fc9c55248af538d84a3c0a526c84f5512e3" Oct 30 22:05:23.853762 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:05:23.853736 5748 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"648bf1c437fe90e4c4cbe008c5c76fc9c55248af538d84a3c0a526c84f5512e3\": container with ID starting with 648bf1c437fe90e4c4cbe008c5c76fc9c55248af538d84a3c0a526c84f5512e3 not found: ID does not exist" containerID="648bf1c437fe90e4c4cbe008c5c76fc9c55248af538d84a3c0a526c84f5512e3" Oct 30 22:05:23.853830 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:23.853779 5748 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"648bf1c437fe90e4c4cbe008c5c76fc9c55248af538d84a3c0a526c84f5512e3"} err="failed to get container status \"648bf1c437fe90e4c4cbe008c5c76fc9c55248af538d84a3c0a526c84f5512e3\": rpc error: code = NotFound desc = could not find container \"648bf1c437fe90e4c4cbe008c5c76fc9c55248af538d84a3c0a526c84f5512e3\": container with ID starting with 648bf1c437fe90e4c4cbe008c5c76fc9c55248af538d84a3c0a526c84f5512e3 not found: ID does not exist" Oct 30 22:05:23.853830 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:23.853803 5748 scope.go:122] "RemoveContainer" containerID="486949e0219addc738f87a366f0ea90c243baa9a9c6cf4bf9ce86aed48906f0f" Oct 30 22:05:24.022977 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:24.022591 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d68156/distinct-w3-d68156-kserve-565894fdb4-85t7z"] Oct 30 22:05:24.071446 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:24.071415 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-d68156/distinct-w3-d68156-kserve-565894fdb4-85t7z"] Oct 30 22:05:24.136831 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:24.136795 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d68156/simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk"] Oct 30 22:05:24.143594 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:24.143557 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-d68156/simulator-w3-d68156-kserve-6b9c9b4c76-r9fkk"] Oct 30 22:05:24.444775 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:24.444740 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe" path="/var/lib/kubelet/pods/13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe/volumes" Oct 30 22:05:24.445439 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:24.445378 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db285e9c-ca87-43fa-8f06-69e271e5c2c5" path="/var/lib/kubelet/pods/db285e9c-ca87-43fa-8f06-69e271e5c2c5/volumes" Oct 30 22:05:25.785094 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:25.785042 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-691598/simulator-w0-691598-kserve-76fb5d5bc7-l5gqk" podUID="4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:25.841921 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:25.822070 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ea374/distinct-2-w1-6ea374-kserve-68fb49889-hpg2k" podUID="ab533734-e0d8-4acb-854d-12851a87bc58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:26.153051 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:26.152989 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h" podUID="0a0b0425-2959-4690-9847-38852e167230" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:27.468648 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:27.468608 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d68156-78b685b5dc-pwg7l" podUID="8b269e36-1397-47d9-a349-04304ad783a4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:05:30.147763 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:05:30.147721 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:05:33.069993 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:33.069959 5748 generic.go:356] "Generic (PLEG): container finished" podID="4bb93eca-1d04-412d-aa83-2e1ac581bf1c" containerID="bb89cacb82ca662a89e11bea6558fa3b1b31b3ce407467306e5459cf2b88faea" exitCode=0 Oct 30 22:05:33.070549 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:33.070039 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj" event={"ID":"4bb93eca-1d04-412d-aa83-2e1ac581bf1c","Type":"ContainerDied","Data":"bb89cacb82ca662a89e11bea6558fa3b1b31b3ce407467306e5459cf2b88faea"} Oct 30 22:05:33.070549 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:33.070535 5748 scope.go:122] "RemoveContainer" containerID="bb89cacb82ca662a89e11bea6558fa3b1b31b3ce407467306e5459cf2b88faea" Oct 30 22:05:33.072047 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:33.072020 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:05:33.072113 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:33.072091 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:05:33.072150 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:33.072124 5748 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 22:05:34.077878 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:34.077836 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj" event={"ID":"4bb93eca-1d04-412d-aa83-2e1ac581bf1c","Type":"ContainerStarted","Data":"4865d5a0e317b2a2978182c6b17a766b0ffab2a5f1471504f1d6aa2e337e9d2c"} Oct 30 22:05:34.222917 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:34.202804 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d979ec1-rkz69"] Oct 30 22:05:34.224956 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:34.223989 5748 cpu_manager.go:414] "containerMap: removing container" podUID="db285e9c-ca87-43fa-8f06-69e271e5c2c5" containerName="main" Oct 30 22:05:34.225136 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:34.224966 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db285e9c-ca87-43fa-8f06-69e271e5c2c5" containerName="main" Oct 30 22:05:34.225136 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:34.225041 5748 cpu_manager.go:414] "containerMap: removing container" podUID="59df8c0f-237e-4d38-9d2d-0982d5f44743" containerName="revoke-keys" Oct 30 22:05:34.225136 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:34.225052 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59df8c0f-237e-4d38-9d2d-0982d5f44743" containerName="revoke-keys" Oct 30 22:05:34.225136 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:34.225099 5748 cpu_manager.go:414] "containerMap: removing container" podUID="13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe" containerName="main" Oct 30 22:05:34.225136 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:34.225108 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe" containerName="main" Oct 30 22:05:34.225406 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:34.225273 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="13ec4fff-3e1b-42c2-83dd-4d23c6af6cbe" containerName="main" Oct 30 22:05:34.225406 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:34.225296 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="db285e9c-ca87-43fa-8f06-69e271e5c2c5" containerName="main" Oct 30 22:05:34.225406 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:34.225309 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="59df8c0f-237e-4d38-9d2d-0982d5f44743" containerName="revoke-keys" Oct 30 22:05:34.227826 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:34.227806 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d979ec1-rkz69" Oct 30 22:05:34.280148 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:34.280120 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a05e77b5-b247-40a2-b6e3-9f75b57feacb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d979ec1-rkz69\" (UID: \"a05e77b5-b247-40a2-b6e3-9f75b57feacb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d979ec1-rkz69" Oct 30 22:05:34.280279 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:34.280252 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7bnr\" (UniqueName: \"kubernetes.io/projected/a05e77b5-b247-40a2-b6e3-9f75b57feacb-kube-api-access-f7bnr\") pod \"maas-api-revoke-sub-1d979ec1-rkz69\" (UID: \"a05e77b5-b247-40a2-b6e3-9f75b57feacb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d979ec1-rkz69" Oct 30 22:05:34.280323 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:34.280315 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a05e77b5-b247-40a2-b6e3-9f75b57feacb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d979ec1-rkz69\" (UID: \"a05e77b5-b247-40a2-b6e3-9f75b57feacb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d979ec1-rkz69" Oct 30 22:05:34.306402 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:34.306374 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:05:34.307410 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:34.306703 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 22:05:34.307555 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:34.306783 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:05:34.307643 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:34.306823 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:05:34.329656 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:34.326952 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-836152a7-f89tv"] Oct 30 22:05:34.338932 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:34.338865 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d979ec1-rkz69"] Oct 30 22:05:34.339097 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:34.339005 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-836152a7-f89tv" Oct 30 22:05:34.374867 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:34.374831 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-836152a7-f89tv"] Oct 30 22:05:34.381361 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:34.381327 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a05e77b5-b247-40a2-b6e3-9f75b57feacb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d979ec1-rkz69\" (UID: \"a05e77b5-b247-40a2-b6e3-9f75b57feacb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d979ec1-rkz69" Oct 30 22:05:34.381537 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:34.381467 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7bnr\" (UniqueName: \"kubernetes.io/projected/a05e77b5-b247-40a2-b6e3-9f75b57feacb-kube-api-access-f7bnr\") pod \"maas-api-revoke-sub-1d979ec1-rkz69\" (UID: \"a05e77b5-b247-40a2-b6e3-9f75b57feacb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d979ec1-rkz69" Oct 30 22:05:34.381603 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:34.381538 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a05e77b5-b247-40a2-b6e3-9f75b57feacb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d979ec1-rkz69\" (UID: \"a05e77b5-b247-40a2-b6e3-9f75b57feacb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d979ec1-rkz69" Oct 30 22:05:34.382529 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:34.382504 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a05e77b5-b247-40a2-b6e3-9f75b57feacb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d979ec1-rkz69\" (UID: \"a05e77b5-b247-40a2-b6e3-9f75b57feacb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d979ec1-rkz69" Oct 30 22:05:34.384652 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:34.384627 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a05e77b5-b247-40a2-b6e3-9f75b57feacb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d979ec1-rkz69\" (UID: \"a05e77b5-b247-40a2-b6e3-9f75b57feacb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d979ec1-rkz69" Oct 30 22:05:34.447457 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:34.447420 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7bnr\" (UniqueName: \"kubernetes.io/projected/a05e77b5-b247-40a2-b6e3-9f75b57feacb-kube-api-access-f7bnr\") pod \"maas-api-revoke-sub-1d979ec1-rkz69\" (UID: \"a05e77b5-b247-40a2-b6e3-9f75b57feacb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d979ec1-rkz69" Oct 30 22:05:34.482554 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:34.482515 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2jsh\" (UniqueName: \"kubernetes.io/projected/ac81c948-6995-40f2-acbe-ee1919a61e0c-kube-api-access-p2jsh\") pod \"maas-api-revoke-sub-836152a7-f89tv\" (UID: \"ac81c948-6995-40f2-acbe-ee1919a61e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-836152a7-f89tv" Oct 30 22:05:34.482729 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:34.482635 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac81c948-6995-40f2-acbe-ee1919a61e0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-836152a7-f89tv\" (UID: \"ac81c948-6995-40f2-acbe-ee1919a61e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-836152a7-f89tv" Oct 30 22:05:34.482729 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:34.482723 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac81c948-6995-40f2-acbe-ee1919a61e0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-836152a7-f89tv\" (UID: \"ac81c948-6995-40f2-acbe-ee1919a61e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-836152a7-f89tv" Oct 30 22:05:34.543201 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:34.543166 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d979ec1-rkz69" Oct 30 22:05:34.589188 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:34.588968 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2jsh\" (UniqueName: \"kubernetes.io/projected/ac81c948-6995-40f2-acbe-ee1919a61e0c-kube-api-access-p2jsh\") pod \"maas-api-revoke-sub-836152a7-f89tv\" (UID: \"ac81c948-6995-40f2-acbe-ee1919a61e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-836152a7-f89tv" Oct 30 22:05:34.590277 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:34.590246 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac81c948-6995-40f2-acbe-ee1919a61e0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-836152a7-f89tv\" (UID: \"ac81c948-6995-40f2-acbe-ee1919a61e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-836152a7-f89tv" Oct 30 22:05:34.590501 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:34.590423 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac81c948-6995-40f2-acbe-ee1919a61e0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-836152a7-f89tv\" (UID: \"ac81c948-6995-40f2-acbe-ee1919a61e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-836152a7-f89tv" Oct 30 22:05:34.590697 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:34.590679 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac81c948-6995-40f2-acbe-ee1919a61e0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-836152a7-f89tv\" (UID: \"ac81c948-6995-40f2-acbe-ee1919a61e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-836152a7-f89tv" Oct 30 22:05:34.597406 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:34.597382 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac81c948-6995-40f2-acbe-ee1919a61e0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-836152a7-f89tv\" (UID: \"ac81c948-6995-40f2-acbe-ee1919a61e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-836152a7-f89tv" Oct 30 22:05:34.653968 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:34.653060 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2jsh\" (UniqueName: \"kubernetes.io/projected/ac81c948-6995-40f2-acbe-ee1919a61e0c-kube-api-access-p2jsh\") pod \"maas-api-revoke-sub-836152a7-f89tv\" (UID: \"ac81c948-6995-40f2-acbe-ee1919a61e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-836152a7-f89tv" Oct 30 22:05:34.656792 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:34.656760 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-836152a7-f89tv" Oct 30 22:05:34.874086 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:34.872640 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj" Oct 30 22:05:34.874086 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:34.873576 5748 patch_prober.go:29] interesting pod/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Oct 30 22:05:34.874086 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:34.873619 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj" podUID="4bb93eca-1d04-412d-aa83-2e1ac581bf1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Oct 30 22:05:35.014922 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:35.012409 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d979ec1-rkz69"] Oct 30 22:05:35.084497 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:35.084464 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d979ec1-rkz69" event={"ID":"a05e77b5-b247-40a2-b6e3-9f75b57feacb","Type":"ContainerStarted","Data":"c7f699c58edb88052696c9fd62b7e4a30b16a9ba423ab5b9966b1bc5cfbc80eb"} Oct 30 22:05:35.231397 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:35.231373 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-836152a7-f89tv"] Oct 30 22:05:35.237332 ip-10-0-143-61 kubenswrapper[5748]: W1030 22:05:35.237302 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac81c948_6995_40f2_acbe_ee1919a61e0c.slice/crio-5c6f4e68ea196e94d770712fd2a3b49f825376dbcacecb7d3f0644e93dfb6228 WatchSource:0}: Error finding container 5c6f4e68ea196e94d770712fd2a3b49f825376dbcacecb7d3f0644e93dfb6228: Status 404 returned error can't find the container with id 5c6f4e68ea196e94d770712fd2a3b49f825376dbcacecb7d3f0644e93dfb6228 Oct 30 22:05:35.826390 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:35.826290 5748 patch_prober.go:29] interesting pod/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Oct 30 22:05:35.826390 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:35.826354 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj" podUID="4bb93eca-1d04-412d-aa83-2e1ac581bf1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Oct 30 22:05:36.091966 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:36.091870 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-836152a7-f89tv" event={"ID":"ac81c948-6995-40f2-acbe-ee1919a61e0c","Type":"ContainerStarted","Data":"fe83cc9d147468c556219123b06f6ae8ba24ce21f3b201f2efcf45b470e2896b"} Oct 30 22:05:36.091966 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:36.091958 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-836152a7-f89tv" event={"ID":"ac81c948-6995-40f2-acbe-ee1919a61e0c","Type":"ContainerStarted","Data":"5c6f4e68ea196e94d770712fd2a3b49f825376dbcacecb7d3f0644e93dfb6228"} Oct 30 22:05:36.093486 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:36.093459 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d979ec1-rkz69" event={"ID":"a05e77b5-b247-40a2-b6e3-9f75b57feacb","Type":"ContainerStarted","Data":"cbdaaa39def44633990a6b78e2a3c18b37dc37b8f37a78bc025c946099f84485"} Oct 30 22:05:36.346933 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:36.346784 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-836152a7-f89tv" podStartSLOduration=2.346768229 podStartE2EDuration="2.346768229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:36.226684418 +0000 UTC m=+2519.969205354" watchObservedRunningTime="2026-10-30 22:05:36.346768229 +0000 UTC m=+2520.089289164" Oct 30 22:05:36.347603 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:36.347533 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d979ec1-rkz69" podStartSLOduration=2.347519949 podStartE2EDuration="2.347519949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:36.344825132 +0000 UTC m=+2520.087346067" watchObservedRunningTime="2026-10-30 22:05:36.347519949 +0000 UTC m=+2520.090040899" Oct 30 22:05:36.527553 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:36.527522 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8c5350-fc5mw"] Oct 30 22:05:36.528940 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:36.528913 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8c5350-fc5mw"] Oct 30 22:05:36.825072 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:36.825023 5748 patch_prober.go:29] interesting pod/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Oct 30 22:05:36.825279 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:36.825084 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj" podUID="4bb93eca-1d04-412d-aa83-2e1ac581bf1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Oct 30 22:05:37.126290 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:37.126202 5748 generic.go:356] "Generic (PLEG): container finished" podID="a05e77b5-b247-40a2-b6e3-9f75b57feacb" containerID="cbdaaa39def44633990a6b78e2a3c18b37dc37b8f37a78bc025c946099f84485" exitCode=0 Oct 30 22:05:37.126706 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:37.126300 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d979ec1-rkz69" event={"ID":"a05e77b5-b247-40a2-b6e3-9f75b57feacb","Type":"ContainerDied","Data":"cbdaaa39def44633990a6b78e2a3c18b37dc37b8f37a78bc025c946099f84485"} Oct 30 22:05:37.128173 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:37.128146 5748 generic.go:356] "Generic (PLEG): container finished" podID="ac81c948-6995-40f2-acbe-ee1919a61e0c" containerID="fe83cc9d147468c556219123b06f6ae8ba24ce21f3b201f2efcf45b470e2896b" exitCode=0 Oct 30 22:05:37.128308 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:37.128206 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-836152a7-f89tv" event={"ID":"ac81c948-6995-40f2-acbe-ee1919a61e0c","Type":"ContainerDied","Data":"fe83cc9d147468c556219123b06f6ae8ba24ce21f3b201f2efcf45b470e2896b"} Oct 30 22:05:37.468606 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:37.468569 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d68156-78b685b5dc-pwg7l" podUID="8b269e36-1397-47d9-a349-04304ad783a4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:05:37.771497 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:37.771413 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj" Oct 30 22:05:37.821458 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:37.821398 5748 patch_prober.go:29] interesting pod/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Oct 30 22:05:37.821723 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:37.821694 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj" podUID="4bb93eca-1d04-412d-aa83-2e1ac581bf1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Oct 30 22:05:38.127687 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:38.127608 5748 scope.go:122] "RemoveContainer" containerID="4e1d1dbc36a8b295e0a60dacf00b47d78272698d2cfbf8f13b7fffc22061883a" Oct 30 22:05:38.137855 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:38.137831 5748 scope.go:122] "RemoveContainer" containerID="7af23afeef1c9d6eb1dea97dbe82176aa08b1586a6163c9d7c157a57ffa8168d" Oct 30 22:05:38.343972 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:38.343662 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-836152a7-f89tv" Oct 30 22:05:38.375472 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:38.375449 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d979ec1-rkz69" Oct 30 22:05:38.432201 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:38.432175 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84255ff8-1706-49bc-aefa-f2c74bdfe0f1" path="/var/lib/kubelet/pods/84255ff8-1706-49bc-aefa-f2c74bdfe0f1/volumes" Oct 30 22:05:38.457743 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:38.457719 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac81c948-6995-40f2-acbe-ee1919a61e0c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac81c948-6995-40f2-acbe-ee1919a61e0c-maas-api-openshift-ca\") pod \"ac81c948-6995-40f2-acbe-ee1919a61e0c\" (UID: \"ac81c948-6995-40f2-acbe-ee1919a61e0c\") " Oct 30 22:05:38.457870 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:38.457856 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a05e77b5-b247-40a2-b6e3-9f75b57feacb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a05e77b5-b247-40a2-b6e3-9f75b57feacb-maas-api-xks-ca\") pod \"a05e77b5-b247-40a2-b6e3-9f75b57feacb\" (UID: \"a05e77b5-b247-40a2-b6e3-9f75b57feacb\") " Oct 30 22:05:38.457993 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:38.457947 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac81c948-6995-40f2-acbe-ee1919a61e0c-kube-api-access-p2jsh\" (UniqueName: \"kubernetes.io/projected/ac81c948-6995-40f2-acbe-ee1919a61e0c-kube-api-access-p2jsh\") pod \"ac81c948-6995-40f2-acbe-ee1919a61e0c\" (UID: \"ac81c948-6995-40f2-acbe-ee1919a61e0c\") " Oct 30 22:05:38.458078 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:38.458060 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a05e77b5-b247-40a2-b6e3-9f75b57feacb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a05e77b5-b247-40a2-b6e3-9f75b57feacb-maas-api-openshift-ca\") pod \"a05e77b5-b247-40a2-b6e3-9f75b57feacb\" (UID: \"a05e77b5-b247-40a2-b6e3-9f75b57feacb\") " Oct 30 22:05:38.458119 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:38.458069 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac81c948-6995-40f2-acbe-ee1919a61e0c-maas-api-openshift-ca" pod "ac81c948-6995-40f2-acbe-ee1919a61e0c" (UID: "ac81c948-6995-40f2-acbe-ee1919a61e0c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:05:38.458119 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:38.458092 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a05e77b5-b247-40a2-b6e3-9f75b57feacb-kube-api-access-f7bnr\" (UniqueName: \"kubernetes.io/projected/a05e77b5-b247-40a2-b6e3-9f75b57feacb-kube-api-access-f7bnr\") pod \"a05e77b5-b247-40a2-b6e3-9f75b57feacb\" (UID: \"a05e77b5-b247-40a2-b6e3-9f75b57feacb\") " Oct 30 22:05:38.458198 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:38.458128 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac81c948-6995-40f2-acbe-ee1919a61e0c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac81c948-6995-40f2-acbe-ee1919a61e0c-maas-api-xks-ca\") pod \"ac81c948-6995-40f2-acbe-ee1919a61e0c\" (UID: \"ac81c948-6995-40f2-acbe-ee1919a61e0c\") " Oct 30 22:05:38.458424 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:38.458401 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a05e77b5-b247-40a2-b6e3-9f75b57feacb-maas-api-openshift-ca" pod "a05e77b5-b247-40a2-b6e3-9f75b57feacb" (UID: "a05e77b5-b247-40a2-b6e3-9f75b57feacb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:05:38.458578 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:38.458555 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac81c948-6995-40f2-acbe-ee1919a61e0c-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:38.458651 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:38.458582 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a05e77b5-b247-40a2-b6e3-9f75b57feacb-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:38.460288 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:38.460265 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac81c948-6995-40f2-acbe-ee1919a61e0c-kube-api-access-p2jsh" pod "ac81c948-6995-40f2-acbe-ee1919a61e0c" (UID: "ac81c948-6995-40f2-acbe-ee1919a61e0c"). InnerVolumeSpecName "kube-api-access-p2jsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:05:38.460376 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:38.460284 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a05e77b5-b247-40a2-b6e3-9f75b57feacb-maas-api-xks-ca" pod "a05e77b5-b247-40a2-b6e3-9f75b57feacb" (UID: "a05e77b5-b247-40a2-b6e3-9f75b57feacb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:05:38.460434 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:38.460412 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a05e77b5-b247-40a2-b6e3-9f75b57feacb-kube-api-access-f7bnr" pod "a05e77b5-b247-40a2-b6e3-9f75b57feacb" (UID: "a05e77b5-b247-40a2-b6e3-9f75b57feacb"). InnerVolumeSpecName "kube-api-access-f7bnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:05:38.460554 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:38.460528 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac81c948-6995-40f2-acbe-ee1919a61e0c-maas-api-xks-ca" pod "ac81c948-6995-40f2-acbe-ee1919a61e0c" (UID: "ac81c948-6995-40f2-acbe-ee1919a61e0c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:05:38.559876 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:38.559841 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a05e77b5-b247-40a2-b6e3-9f75b57feacb-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:38.559876 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:38.559875 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2jsh\" (UniqueName: \"kubernetes.io/projected/ac81c948-6995-40f2-acbe-ee1919a61e0c-kube-api-access-p2jsh\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:38.559876 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:38.559885 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7bnr\" (UniqueName: \"kubernetes.io/projected/a05e77b5-b247-40a2-b6e3-9f75b57feacb-kube-api-access-f7bnr\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:38.559876 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:38.559906 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac81c948-6995-40f2-acbe-ee1919a61e0c-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:38.825212 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:38.825087 5748 patch_prober.go:29] interesting pod/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Oct 30 22:05:38.825212 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:38.825172 5748 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj" podUID="4bb93eca-1d04-412d-aa83-2e1ac581bf1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Oct 30 22:05:39.141269 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:39.141234 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-836152a7-f89tv" event={"ID":"ac81c948-6995-40f2-acbe-ee1919a61e0c","Type":"ContainerDied","Data":"5c6f4e68ea196e94d770712fd2a3b49f825376dbcacecb7d3f0644e93dfb6228"} Oct 30 22:05:39.141653 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:39.141284 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c6f4e68ea196e94d770712fd2a3b49f825376dbcacecb7d3f0644e93dfb6228" Oct 30 22:05:39.141653 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:39.141332 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-836152a7-f89tv" Oct 30 22:05:39.145072 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:39.145050 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d979ec1-rkz69" Oct 30 22:05:39.145072 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:39.145054 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d979ec1-rkz69" event={"ID":"a05e77b5-b247-40a2-b6e3-9f75b57feacb","Type":"ContainerDied","Data":"c7f699c58edb88052696c9fd62b7e4a30b16a9ba423ab5b9966b1bc5cfbc80eb"} Oct 30 22:05:39.145219 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:39.145103 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7f699c58edb88052696c9fd62b7e4a30b16a9ba423ab5b9966b1bc5cfbc80eb" Oct 30 22:05:39.824201 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:39.824171 5748 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj" Oct 30 22:05:39.825010 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:39.824995 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj" Oct 30 22:05:40.153040 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:40.153011 5748 generic.go:356] "Generic (PLEG): container finished" podID="8b269e36-1397-47d9-a349-04304ad783a4" containerID="5560debf400a8ca000f0638c00abd098dd70e6616236a15d7727409a3fd3d78f" exitCode=0 Oct 30 22:05:40.153432 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:40.153081 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d68156-78b685b5dc-pwg7l" event={"ID":"8b269e36-1397-47d9-a349-04304ad783a4","Type":"ContainerDied","Data":"5560debf400a8ca000f0638c00abd098dd70e6616236a15d7727409a3fd3d78f"} Oct 30 22:05:40.289062 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:40.289039 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d68156-78b685b5dc-pwg7l" Oct 30 22:05:40.382229 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:40.382195 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b269e36-1397-47d9-a349-04304ad783a4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b269e36-1397-47d9-a349-04304ad783a4-plugins-config-volume\") pod \"8b269e36-1397-47d9-a349-04304ad783a4\" (UID: \"8b269e36-1397-47d9-a349-04304ad783a4\") " Oct 30 22:05:40.382382 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:40.382318 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b269e36-1397-47d9-a349-04304ad783a4-kube-api-access-lqlnf\" (UniqueName: \"kubernetes.io/projected/8b269e36-1397-47d9-a349-04304ad783a4-kube-api-access-lqlnf\") pod \"8b269e36-1397-47d9-a349-04304ad783a4\" (UID: \"8b269e36-1397-47d9-a349-04304ad783a4\") " Oct 30 22:05:40.382594 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:40.382570 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b269e36-1397-47d9-a349-04304ad783a4-plugins-config-volume" pod "8b269e36-1397-47d9-a349-04304ad783a4" (UID: "8b269e36-1397-47d9-a349-04304ad783a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:05:40.384427 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:40.384404 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b269e36-1397-47d9-a349-04304ad783a4-kube-api-access-lqlnf" pod "8b269e36-1397-47d9-a349-04304ad783a4" (UID: "8b269e36-1397-47d9-a349-04304ad783a4"). InnerVolumeSpecName "kube-api-access-lqlnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:05:40.483833 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:40.483801 5748 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b269e36-1397-47d9-a349-04304ad783a4-plugins-config-volume\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:40.483833 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:40.483829 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqlnf\" (UniqueName: \"kubernetes.io/projected/8b269e36-1397-47d9-a349-04304ad783a4-kube-api-access-lqlnf\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:41.160125 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:41.160093 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d68156-78b685b5dc-pwg7l" Oct 30 22:05:41.160564 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:41.160135 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d68156-78b685b5dc-pwg7l" event={"ID":"8b269e36-1397-47d9-a349-04304ad783a4","Type":"ContainerDied","Data":"469fbb88cb9f7ed76ce0ccb70520c01377000ae07791fdac3a3821158b8634a4"} Oct 30 22:05:41.160564 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:41.160211 5748 scope.go:122] "RemoveContainer" containerID="5560debf400a8ca000f0638c00abd098dd70e6616236a15d7727409a3fd3d78f" Oct 30 22:05:41.327418 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:41.327384 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-d68156-78b685b5dc-pwg7l"] Oct 30 22:05:41.377798 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:41.377766 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-d68156-78b685b5dc-pwg7l"] Oct 30 22:05:42.416275 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:42.416197 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b269e36-1397-47d9-a349-04304ad783a4" path="/var/lib/kubelet/pods/8b269e36-1397-47d9-a349-04304ad783a4/volumes" Oct 30 22:05:54.507883 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:54.507849 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-abb526-bgjz6"] Oct 30 22:05:54.508365 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:54.508349 5748 cpu_manager.go:414] "containerMap: removing container" podUID="ac81c948-6995-40f2-acbe-ee1919a61e0c" containerName="revoke-keys" Oct 30 22:05:54.508412 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:54.508366 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac81c948-6995-40f2-acbe-ee1919a61e0c" containerName="revoke-keys" Oct 30 22:05:54.508412 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:54.508375 5748 cpu_manager.go:414] "containerMap: removing container" podUID="8b269e36-1397-47d9-a349-04304ad783a4" containerName="payload-pre-processing" Oct 30 22:05:54.508412 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:54.508381 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b269e36-1397-47d9-a349-04304ad783a4" containerName="payload-pre-processing" Oct 30 22:05:54.508412 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:54.508405 5748 cpu_manager.go:414] "containerMap: removing container" podUID="a05e77b5-b247-40a2-b6e3-9f75b57feacb" containerName="revoke-keys" Oct 30 22:05:54.508412 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:54.508410 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a05e77b5-b247-40a2-b6e3-9f75b57feacb" containerName="revoke-keys" Oct 30 22:05:54.508564 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:54.508512 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac81c948-6995-40f2-acbe-ee1919a61e0c" containerName="revoke-keys" Oct 30 22:05:54.508564 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:54.508522 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b269e36-1397-47d9-a349-04304ad783a4" containerName="payload-pre-processing" Oct 30 22:05:54.508564 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:54.508531 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="a05e77b5-b247-40a2-b6e3-9f75b57feacb" containerName="revoke-keys" Oct 30 22:05:54.511385 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:54.511369 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-abb526-bgjz6" Oct 30 22:05:54.559307 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:54.559278 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39265b00-5066-4f25-a753-8d27e5e7dc3c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-abb526-bgjz6\" (UID: \"39265b00-5066-4f25-a753-8d27e5e7dc3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-abb526-bgjz6" Oct 30 22:05:54.559441 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:54.559346 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39265b00-5066-4f25-a753-8d27e5e7dc3c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-abb526-bgjz6\" (UID: \"39265b00-5066-4f25-a753-8d27e5e7dc3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-abb526-bgjz6" Oct 30 22:05:54.559486 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:54.559454 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8knr2\" (UniqueName: \"kubernetes.io/projected/39265b00-5066-4f25-a753-8d27e5e7dc3c-kube-api-access-8knr2\") pod \"maas-api-revoke-keys-e2e-inf-b-abb526-bgjz6\" (UID: \"39265b00-5066-4f25-a753-8d27e5e7dc3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-abb526-bgjz6" Oct 30 22:05:54.567128 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:54.567100 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:05:54.617295 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:54.617266 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:05:54.617469 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:54.617317 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:05:54.617469 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:54.617402 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 22:05:54.620237 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:54.620210 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-abb526-bgjz6"] Oct 30 22:05:54.660906 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:54.660833 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39265b00-5066-4f25-a753-8d27e5e7dc3c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-abb526-bgjz6\" (UID: \"39265b00-5066-4f25-a753-8d27e5e7dc3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-abb526-bgjz6" Oct 30 22:05:54.661049 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:54.660908 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39265b00-5066-4f25-a753-8d27e5e7dc3c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-abb526-bgjz6\" (UID: \"39265b00-5066-4f25-a753-8d27e5e7dc3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-abb526-bgjz6" Oct 30 22:05:54.661049 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:54.660956 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8knr2\" (UniqueName: \"kubernetes.io/projected/39265b00-5066-4f25-a753-8d27e5e7dc3c-kube-api-access-8knr2\") pod \"maas-api-revoke-keys-e2e-inf-b-abb526-bgjz6\" (UID: \"39265b00-5066-4f25-a753-8d27e5e7dc3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-abb526-bgjz6" Oct 30 22:05:54.661481 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:54.661463 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39265b00-5066-4f25-a753-8d27e5e7dc3c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-abb526-bgjz6\" (UID: \"39265b00-5066-4f25-a753-8d27e5e7dc3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-abb526-bgjz6" Oct 30 22:05:54.663414 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:54.663387 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39265b00-5066-4f25-a753-8d27e5e7dc3c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-abb526-bgjz6\" (UID: \"39265b00-5066-4f25-a753-8d27e5e7dc3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-abb526-bgjz6" Oct 30 22:05:54.722736 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:54.722715 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8knr2\" (UniqueName: \"kubernetes.io/projected/39265b00-5066-4f25-a753-8d27e5e7dc3c-kube-api-access-8knr2\") pod \"maas-api-revoke-keys-e2e-inf-b-abb526-bgjz6\" (UID: \"39265b00-5066-4f25-a753-8d27e5e7dc3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-abb526-bgjz6" Oct 30 22:05:54.823256 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:54.823224 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-abb526-bgjz6" Oct 30 22:05:55.200439 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:55.200414 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-abb526-bgjz6"] Oct 30 22:05:55.202317 ip-10-0-143-61 kubenswrapper[5748]: W1030 22:05:55.202287 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39265b00_5066_4f25_a753_8d27e5e7dc3c.slice/crio-cc42d2697486b75fb7731fc9aa1498bc53e5d26f84b6081f281ef04d3e609246 WatchSource:0}: Error finding container cc42d2697486b75fb7731fc9aa1498bc53e5d26f84b6081f281ef04d3e609246: Status 404 returned error can't find the container with id cc42d2697486b75fb7731fc9aa1498bc53e5d26f84b6081f281ef04d3e609246 Oct 30 22:05:55.243991 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:55.243961 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-abb526-bgjz6" event={"ID":"39265b00-5066-4f25-a753-8d27e5e7dc3c","Type":"ContainerStarted","Data":"cc42d2697486b75fb7731fc9aa1498bc53e5d26f84b6081f281ef04d3e609246"} Oct 30 22:05:56.251592 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:56.251553 5748 generic.go:356] "Generic (PLEG): container finished" podID="39265b00-5066-4f25-a753-8d27e5e7dc3c" containerID="93fb0d7a3eb48102b4bc96a8b599df04eae9664c37f3ec7dc9dbb2a89a48f52f" exitCode=0 Oct 30 22:05:56.251986 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:56.251599 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-abb526-bgjz6" event={"ID":"39265b00-5066-4f25-a753-8d27e5e7dc3c","Type":"ContainerDied","Data":"93fb0d7a3eb48102b4bc96a8b599df04eae9664c37f3ec7dc9dbb2a89a48f52f"} Oct 30 22:05:57.420572 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:57.420548 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-abb526-bgjz6" Oct 30 22:05:57.497050 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:57.496189 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39265b00-5066-4f25-a753-8d27e5e7dc3c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39265b00-5066-4f25-a753-8d27e5e7dc3c-maas-api-xks-ca\") pod \"39265b00-5066-4f25-a753-8d27e5e7dc3c\" (UID: \"39265b00-5066-4f25-a753-8d27e5e7dc3c\") " Oct 30 22:05:57.497050 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:57.496265 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39265b00-5066-4f25-a753-8d27e5e7dc3c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39265b00-5066-4f25-a753-8d27e5e7dc3c-maas-api-openshift-ca\") pod \"39265b00-5066-4f25-a753-8d27e5e7dc3c\" (UID: \"39265b00-5066-4f25-a753-8d27e5e7dc3c\") " Oct 30 22:05:57.497050 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:57.496440 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39265b00-5066-4f25-a753-8d27e5e7dc3c-kube-api-access-8knr2\" (UniqueName: \"kubernetes.io/projected/39265b00-5066-4f25-a753-8d27e5e7dc3c-kube-api-access-8knr2\") pod \"39265b00-5066-4f25-a753-8d27e5e7dc3c\" (UID: \"39265b00-5066-4f25-a753-8d27e5e7dc3c\") " Oct 30 22:05:57.497050 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:57.496963 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39265b00-5066-4f25-a753-8d27e5e7dc3c-maas-api-openshift-ca" pod "39265b00-5066-4f25-a753-8d27e5e7dc3c" (UID: "39265b00-5066-4f25-a753-8d27e5e7dc3c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:05:57.500303 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:57.500267 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39265b00-5066-4f25-a753-8d27e5e7dc3c-maas-api-xks-ca" pod "39265b00-5066-4f25-a753-8d27e5e7dc3c" (UID: "39265b00-5066-4f25-a753-8d27e5e7dc3c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:05:57.502848 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:57.502822 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39265b00-5066-4f25-a753-8d27e5e7dc3c-kube-api-access-8knr2" pod "39265b00-5066-4f25-a753-8d27e5e7dc3c" (UID: "39265b00-5066-4f25-a753-8d27e5e7dc3c"). InnerVolumeSpecName "kube-api-access-8knr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:05:57.598314 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:57.598284 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39265b00-5066-4f25-a753-8d27e5e7dc3c-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:57.598314 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:57.598314 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39265b00-5066-4f25-a753-8d27e5e7dc3c-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:57.598523 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:57.598325 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8knr2\" (UniqueName: \"kubernetes.io/projected/39265b00-5066-4f25-a753-8d27e5e7dc3c-kube-api-access-8knr2\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:58.264839 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:58.264810 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-abb526-bgjz6" Oct 30 22:05:58.265032 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:58.264848 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-abb526-bgjz6" event={"ID":"39265b00-5066-4f25-a753-8d27e5e7dc3c","Type":"ContainerDied","Data":"cc42d2697486b75fb7731fc9aa1498bc53e5d26f84b6081f281ef04d3e609246"} Oct 30 22:05:58.265032 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:05:58.264915 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc42d2697486b75fb7731fc9aa1498bc53e5d26f84b6081f281ef04d3e609246" Oct 30 22:06:00.144851 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:06:00.144813 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:06:12.728685 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:12.728649 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-98d38f-kh4cd"] Oct 30 22:06:12.729189 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:12.729174 5748 cpu_manager.go:414] "containerMap: removing container" podUID="39265b00-5066-4f25-a753-8d27e5e7dc3c" containerName="revoke-keys" Oct 30 22:06:12.729189 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:12.729190 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39265b00-5066-4f25-a753-8d27e5e7dc3c" containerName="revoke-keys" Oct 30 22:06:12.729316 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:12.729285 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="39265b00-5066-4f25-a753-8d27e5e7dc3c" containerName="revoke-keys" Oct 30 22:06:12.732885 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:12.732868 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-98d38f-kh4cd" Oct 30 22:06:12.792443 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:12.792411 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-98d38f-kh4cd"] Oct 30 22:06:12.839013 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:12.838972 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:06:12.839013 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:12.838981 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:06:12.839290 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:12.838981 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:06:12.839290 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:12.839057 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 22:06:12.884034 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:12.884003 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06e13f56-0557-4d59-b085-5a195344e67e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-98d38f-kh4cd\" (UID: \"06e13f56-0557-4d59-b085-5a195344e67e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-98d38f-kh4cd" Oct 30 22:06:12.884185 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:12.884059 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn9t8\" (UniqueName: \"kubernetes.io/projected/06e13f56-0557-4d59-b085-5a195344e67e-kube-api-access-nn9t8\") pod \"maas-api-revoke-keys-e2e-api-a-98d38f-kh4cd\" (UID: \"06e13f56-0557-4d59-b085-5a195344e67e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-98d38f-kh4cd" Oct 30 22:06:12.884185 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:12.884167 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06e13f56-0557-4d59-b085-5a195344e67e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-98d38f-kh4cd\" (UID: \"06e13f56-0557-4d59-b085-5a195344e67e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-98d38f-kh4cd" Oct 30 22:06:12.986090 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:12.985996 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06e13f56-0557-4d59-b085-5a195344e67e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-98d38f-kh4cd\" (UID: \"06e13f56-0557-4d59-b085-5a195344e67e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-98d38f-kh4cd" Oct 30 22:06:12.986250 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:12.986142 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06e13f56-0557-4d59-b085-5a195344e67e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-98d38f-kh4cd\" (UID: \"06e13f56-0557-4d59-b085-5a195344e67e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-98d38f-kh4cd" Oct 30 22:06:12.986250 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:12.986180 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn9t8\" (UniqueName: \"kubernetes.io/projected/06e13f56-0557-4d59-b085-5a195344e67e-kube-api-access-nn9t8\") pod \"maas-api-revoke-keys-e2e-api-a-98d38f-kh4cd\" (UID: \"06e13f56-0557-4d59-b085-5a195344e67e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-98d38f-kh4cd" Oct 30 22:06:12.986668 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:12.986645 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06e13f56-0557-4d59-b085-5a195344e67e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-98d38f-kh4cd\" (UID: \"06e13f56-0557-4d59-b085-5a195344e67e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-98d38f-kh4cd" Oct 30 22:06:12.988436 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:12.988416 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06e13f56-0557-4d59-b085-5a195344e67e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-98d38f-kh4cd\" (UID: \"06e13f56-0557-4d59-b085-5a195344e67e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-98d38f-kh4cd" Oct 30 22:06:13.047549 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:13.047520 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn9t8\" (UniqueName: \"kubernetes.io/projected/06e13f56-0557-4d59-b085-5a195344e67e-kube-api-access-nn9t8\") pod \"maas-api-revoke-keys-e2e-api-a-98d38f-kh4cd\" (UID: \"06e13f56-0557-4d59-b085-5a195344e67e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-98d38f-kh4cd" Oct 30 22:06:13.347300 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:13.347220 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-98d38f-kh4cd" Oct 30 22:06:13.724269 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:13.724239 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-98d38f-kh4cd"] Oct 30 22:06:13.725981 ip-10-0-143-61 kubenswrapper[5748]: W1030 22:06:13.725951 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06e13f56_0557_4d59_b085_5a195344e67e.slice/crio-8d4c97eb5f94e197b28c98590c016d4dd2106e906b40082887e43826401b0118 WatchSource:0}: Error finding container 8d4c97eb5f94e197b28c98590c016d4dd2106e906b40082887e43826401b0118: Status 404 returned error can't find the container with id 8d4c97eb5f94e197b28c98590c016d4dd2106e906b40082887e43826401b0118 Oct 30 22:06:14.366088 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:14.365995 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-98d38f-kh4cd" event={"ID":"06e13f56-0557-4d59-b085-5a195344e67e","Type":"ContainerStarted","Data":"caef26ad7fb09979be58ff2182e48c7b9170d24e5fc4b8f531db174df8790935"} Oct 30 22:06:14.366088 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:14.366045 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-98d38f-kh4cd" event={"ID":"06e13f56-0557-4d59-b085-5a195344e67e","Type":"ContainerStarted","Data":"8d4c97eb5f94e197b28c98590c016d4dd2106e906b40082887e43826401b0118"} Oct 30 22:06:14.484213 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:14.484163 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-98d38f-kh4cd" podStartSLOduration=2.484146945 podStartE2EDuration="2.484146945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:14.484098895 +0000 UTC m=+2558.226619831" watchObservedRunningTime="2026-10-30 22:06:14.484146945 +0000 UTC m=+2558.226667879" Oct 30 22:06:15.374218 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:15.374188 5748 generic.go:356] "Generic (PLEG): container finished" podID="06e13f56-0557-4d59-b085-5a195344e67e" containerID="caef26ad7fb09979be58ff2182e48c7b9170d24e5fc4b8f531db174df8790935" exitCode=0 Oct 30 22:06:15.374627 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:15.374275 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-98d38f-kh4cd" event={"ID":"06e13f56-0557-4d59-b085-5a195344e67e","Type":"ContainerDied","Data":"caef26ad7fb09979be58ff2182e48c7b9170d24e5fc4b8f531db174df8790935"} Oct 30 22:06:16.545232 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:16.545208 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-98d38f-kh4cd" Oct 30 22:06:16.731116 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:16.731042 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06e13f56-0557-4d59-b085-5a195344e67e-kube-api-access-nn9t8\" (UniqueName: \"kubernetes.io/projected/06e13f56-0557-4d59-b085-5a195344e67e-kube-api-access-nn9t8\") pod \"06e13f56-0557-4d59-b085-5a195344e67e\" (UID: \"06e13f56-0557-4d59-b085-5a195344e67e\") " Oct 30 22:06:16.731270 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:16.731168 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06e13f56-0557-4d59-b085-5a195344e67e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06e13f56-0557-4d59-b085-5a195344e67e-maas-api-xks-ca\") pod \"06e13f56-0557-4d59-b085-5a195344e67e\" (UID: \"06e13f56-0557-4d59-b085-5a195344e67e\") " Oct 30 22:06:16.731270 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:16.731230 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06e13f56-0557-4d59-b085-5a195344e67e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06e13f56-0557-4d59-b085-5a195344e67e-maas-api-openshift-ca\") pod \"06e13f56-0557-4d59-b085-5a195344e67e\" (UID: \"06e13f56-0557-4d59-b085-5a195344e67e\") " Oct 30 22:06:16.731591 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:16.731552 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06e13f56-0557-4d59-b085-5a195344e67e-maas-api-openshift-ca" pod "06e13f56-0557-4d59-b085-5a195344e67e" (UID: "06e13f56-0557-4d59-b085-5a195344e67e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:06:16.733133 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:16.733110 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06e13f56-0557-4d59-b085-5a195344e67e-kube-api-access-nn9t8" pod "06e13f56-0557-4d59-b085-5a195344e67e" (UID: "06e13f56-0557-4d59-b085-5a195344e67e"). InnerVolumeSpecName "kube-api-access-nn9t8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:06:16.733280 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:16.733261 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06e13f56-0557-4d59-b085-5a195344e67e-maas-api-xks-ca" pod "06e13f56-0557-4d59-b085-5a195344e67e" (UID: "06e13f56-0557-4d59-b085-5a195344e67e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:06:16.832637 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:16.832607 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nn9t8\" (UniqueName: \"kubernetes.io/projected/06e13f56-0557-4d59-b085-5a195344e67e-kube-api-access-nn9t8\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:06:16.832637 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:16.832634 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06e13f56-0557-4d59-b085-5a195344e67e-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:06:16.832637 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:16.832644 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06e13f56-0557-4d59-b085-5a195344e67e-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:06:17.174497 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:17.174463 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ae1bd5-h9xff"] Oct 30 22:06:17.175063 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:17.175047 5748 cpu_manager.go:414] "containerMap: removing container" podUID="06e13f56-0557-4d59-b085-5a195344e67e" containerName="revoke-keys" Oct 30 22:06:17.175063 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:17.175063 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06e13f56-0557-4d59-b085-5a195344e67e" containerName="revoke-keys" Oct 30 22:06:17.175234 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:17.175152 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="06e13f56-0557-4d59-b085-5a195344e67e" containerName="revoke-keys" Oct 30 22:06:17.178600 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:17.178579 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ae1bd5-h9xff" Oct 30 22:06:17.256409 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:17.256377 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ae1bd5-h9xff"] Oct 30 22:06:17.338054 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:17.338027 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dkgf\" (UniqueName: \"kubernetes.io/projected/237d15fa-293c-4601-aa9e-3f233c758428-kube-api-access-7dkgf\") pod \"maas-api-revoke-keys-e2e-api-a-ae1bd5-h9xff\" (UID: \"237d15fa-293c-4601-aa9e-3f233c758428\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ae1bd5-h9xff" Oct 30 22:06:17.338203 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:17.338081 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/237d15fa-293c-4601-aa9e-3f233c758428-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ae1bd5-h9xff\" (UID: \"237d15fa-293c-4601-aa9e-3f233c758428\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ae1bd5-h9xff" Oct 30 22:06:17.338203 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:17.338176 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/237d15fa-293c-4601-aa9e-3f233c758428-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ae1bd5-h9xff\" (UID: \"237d15fa-293c-4601-aa9e-3f233c758428\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ae1bd5-h9xff" Oct 30 22:06:17.394321 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:17.394290 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-98d38f-kh4cd" Oct 30 22:06:17.394468 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:17.394319 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-98d38f-kh4cd" event={"ID":"06e13f56-0557-4d59-b085-5a195344e67e","Type":"ContainerDied","Data":"8d4c97eb5f94e197b28c98590c016d4dd2106e906b40082887e43826401b0118"} Oct 30 22:06:17.394468 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:17.394373 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d4c97eb5f94e197b28c98590c016d4dd2106e906b40082887e43826401b0118" Oct 30 22:06:17.439404 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:17.439339 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dkgf\" (UniqueName: \"kubernetes.io/projected/237d15fa-293c-4601-aa9e-3f233c758428-kube-api-access-7dkgf\") pod \"maas-api-revoke-keys-e2e-api-a-ae1bd5-h9xff\" (UID: \"237d15fa-293c-4601-aa9e-3f233c758428\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ae1bd5-h9xff" Oct 30 22:06:17.439404 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:17.439392 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/237d15fa-293c-4601-aa9e-3f233c758428-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ae1bd5-h9xff\" (UID: \"237d15fa-293c-4601-aa9e-3f233c758428\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ae1bd5-h9xff" Oct 30 22:06:17.439574 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:17.439428 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/237d15fa-293c-4601-aa9e-3f233c758428-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ae1bd5-h9xff\" (UID: \"237d15fa-293c-4601-aa9e-3f233c758428\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ae1bd5-h9xff" Oct 30 22:06:17.440053 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:17.440029 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/237d15fa-293c-4601-aa9e-3f233c758428-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ae1bd5-h9xff\" (UID: \"237d15fa-293c-4601-aa9e-3f233c758428\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ae1bd5-h9xff" Oct 30 22:06:17.441907 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:17.441871 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/237d15fa-293c-4601-aa9e-3f233c758428-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ae1bd5-h9xff\" (UID: \"237d15fa-293c-4601-aa9e-3f233c758428\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ae1bd5-h9xff" Oct 30 22:06:17.505981 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:17.505948 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dkgf\" (UniqueName: \"kubernetes.io/projected/237d15fa-293c-4601-aa9e-3f233c758428-kube-api-access-7dkgf\") pod \"maas-api-revoke-keys-e2e-api-a-ae1bd5-h9xff\" (UID: \"237d15fa-293c-4601-aa9e-3f233c758428\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ae1bd5-h9xff" Oct 30 22:06:17.793767 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:17.793678 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ae1bd5-h9xff" Oct 30 22:06:18.172334 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:18.172305 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ae1bd5-h9xff"] Oct 30 22:06:18.174421 ip-10-0-143-61 kubenswrapper[5748]: W1030 22:06:18.174394 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod237d15fa_293c_4601_aa9e_3f233c758428.slice/crio-137144b1ec744b2b0e1fcc152821565332eb3b5b881ebc7c13ecff53bcea00c1 WatchSource:0}: Error finding container 137144b1ec744b2b0e1fcc152821565332eb3b5b881ebc7c13ecff53bcea00c1: Status 404 returned error can't find the container with id 137144b1ec744b2b0e1fcc152821565332eb3b5b881ebc7c13ecff53bcea00c1 Oct 30 22:06:18.420392 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:18.420354 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ae1bd5-h9xff" event={"ID":"237d15fa-293c-4601-aa9e-3f233c758428","Type":"ContainerStarted","Data":"137144b1ec744b2b0e1fcc152821565332eb3b5b881ebc7c13ecff53bcea00c1"} Oct 30 22:06:19.423255 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:19.423167 5748 generic.go:356] "Generic (PLEG): container finished" podID="237d15fa-293c-4601-aa9e-3f233c758428" containerID="e8d2fd0cb71df71c55152fd7a66c73eb5f4cc40e8dc5f1b96ab7d7b2361e1292" exitCode=0 Oct 30 22:06:19.423655 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:19.423292 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ae1bd5-h9xff" event={"ID":"237d15fa-293c-4601-aa9e-3f233c758428","Type":"ContainerDied","Data":"e8d2fd0cb71df71c55152fd7a66c73eb5f4cc40e8dc5f1b96ab7d7b2361e1292"} Oct 30 22:06:20.584740 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:20.584714 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ae1bd5-h9xff" Oct 30 22:06:20.673604 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:20.673568 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/237d15fa-293c-4601-aa9e-3f233c758428-kube-api-access-7dkgf\" (UniqueName: \"kubernetes.io/projected/237d15fa-293c-4601-aa9e-3f233c758428-kube-api-access-7dkgf\") pod \"237d15fa-293c-4601-aa9e-3f233c758428\" (UID: \"237d15fa-293c-4601-aa9e-3f233c758428\") " Oct 30 22:06:20.673768 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:20.673633 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/237d15fa-293c-4601-aa9e-3f233c758428-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/237d15fa-293c-4601-aa9e-3f233c758428-maas-api-xks-ca\") pod \"237d15fa-293c-4601-aa9e-3f233c758428\" (UID: \"237d15fa-293c-4601-aa9e-3f233c758428\") " Oct 30 22:06:20.673768 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:20.673711 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/237d15fa-293c-4601-aa9e-3f233c758428-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/237d15fa-293c-4601-aa9e-3f233c758428-maas-api-openshift-ca\") pod \"237d15fa-293c-4601-aa9e-3f233c758428\" (UID: \"237d15fa-293c-4601-aa9e-3f233c758428\") " Oct 30 22:06:20.674131 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:20.674104 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/237d15fa-293c-4601-aa9e-3f233c758428-maas-api-openshift-ca" pod "237d15fa-293c-4601-aa9e-3f233c758428" (UID: "237d15fa-293c-4601-aa9e-3f233c758428"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:06:20.675688 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:20.675659 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/237d15fa-293c-4601-aa9e-3f233c758428-maas-api-xks-ca" pod "237d15fa-293c-4601-aa9e-3f233c758428" (UID: "237d15fa-293c-4601-aa9e-3f233c758428"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:06:20.675786 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:20.675747 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/237d15fa-293c-4601-aa9e-3f233c758428-kube-api-access-7dkgf" pod "237d15fa-293c-4601-aa9e-3f233c758428" (UID: "237d15fa-293c-4601-aa9e-3f233c758428"). InnerVolumeSpecName "kube-api-access-7dkgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:06:20.775680 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:20.775643 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dkgf\" (UniqueName: \"kubernetes.io/projected/237d15fa-293c-4601-aa9e-3f233c758428-kube-api-access-7dkgf\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:06:20.775680 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:20.775674 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/237d15fa-293c-4601-aa9e-3f233c758428-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:06:20.775680 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:20.775684 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/237d15fa-293c-4601-aa9e-3f233c758428-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:06:21.437361 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:21.437325 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ae1bd5-h9xff" event={"ID":"237d15fa-293c-4601-aa9e-3f233c758428","Type":"ContainerDied","Data":"137144b1ec744b2b0e1fcc152821565332eb3b5b881ebc7c13ecff53bcea00c1"} Oct 30 22:06:21.437564 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:21.437364 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ae1bd5-h9xff" Oct 30 22:06:21.437564 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:21.437372 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="137144b1ec744b2b0e1fcc152821565332eb3b5b881ebc7c13ecff53bcea00c1" Oct 30 22:06:24.271776 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:24.271731 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-abb526-bgjz6"] Oct 30 22:06:24.321858 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:24.321826 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-abb526-bgjz6"] Oct 30 22:06:24.416467 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:24.416436 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39265b00-5066-4f25-a753-8d27e5e7dc3c" path="/var/lib/kubelet/pods/39265b00-5066-4f25-a753-8d27e5e7dc3c/volumes" Oct 30 22:06:30.145351 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:06:30.145310 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:06:42.087146 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:42.087104 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2c8f56-9b98j"] Oct 30 22:06:42.087869 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:42.087844 5748 cpu_manager.go:414] "containerMap: removing container" podUID="237d15fa-293c-4601-aa9e-3f233c758428" containerName="revoke-keys" Oct 30 22:06:42.087869 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:42.087870 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="237d15fa-293c-4601-aa9e-3f233c758428" containerName="revoke-keys" Oct 30 22:06:42.088067 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:42.088033 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="237d15fa-293c-4601-aa9e-3f233c758428" containerName="revoke-keys" Oct 30 22:06:42.092883 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:42.092863 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2c8f56-9b98j" Oct 30 22:06:42.149307 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:42.149280 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2c8f56-9b98j"] Oct 30 22:06:42.183455 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:42.183427 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c4fb534-f1fe-4fd2-bc84-35369cddcd1c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-2c8f56-9b98j\" (UID: \"4c4fb534-f1fe-4fd2-bc84-35369cddcd1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2c8f56-9b98j" Oct 30 22:06:42.183599 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:42.183470 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nx7r\" (UniqueName: \"kubernetes.io/projected/4c4fb534-f1fe-4fd2-bc84-35369cddcd1c-kube-api-access-2nx7r\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-2c8f56-9b98j\" (UID: \"4c4fb534-f1fe-4fd2-bc84-35369cddcd1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2c8f56-9b98j" Oct 30 22:06:42.183599 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:42.183488 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c4fb534-f1fe-4fd2-bc84-35369cddcd1c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-2c8f56-9b98j\" (UID: \"4c4fb534-f1fe-4fd2-bc84-35369cddcd1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2c8f56-9b98j" Oct 30 22:06:42.196626 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:42.196598 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:06:42.196752 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:42.196646 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:06:42.196752 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:42.196651 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:06:42.199430 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:42.199405 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 22:06:42.284633 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:42.284597 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c4fb534-f1fe-4fd2-bc84-35369cddcd1c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-2c8f56-9b98j\" (UID: \"4c4fb534-f1fe-4fd2-bc84-35369cddcd1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2c8f56-9b98j" Oct 30 22:06:42.284818 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:42.284656 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nx7r\" (UniqueName: \"kubernetes.io/projected/4c4fb534-f1fe-4fd2-bc84-35369cddcd1c-kube-api-access-2nx7r\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-2c8f56-9b98j\" (UID: \"4c4fb534-f1fe-4fd2-bc84-35369cddcd1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2c8f56-9b98j" Oct 30 22:06:42.284818 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:42.284681 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c4fb534-f1fe-4fd2-bc84-35369cddcd1c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-2c8f56-9b98j\" (UID: \"4c4fb534-f1fe-4fd2-bc84-35369cddcd1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2c8f56-9b98j" Oct 30 22:06:42.285239 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:42.285204 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c4fb534-f1fe-4fd2-bc84-35369cddcd1c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-2c8f56-9b98j\" (UID: \"4c4fb534-f1fe-4fd2-bc84-35369cddcd1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2c8f56-9b98j" Oct 30 22:06:42.287082 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:42.287061 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c4fb534-f1fe-4fd2-bc84-35369cddcd1c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-2c8f56-9b98j\" (UID: \"4c4fb534-f1fe-4fd2-bc84-35369cddcd1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2c8f56-9b98j" Oct 30 22:06:42.346288 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:42.346216 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nx7r\" (UniqueName: \"kubernetes.io/projected/4c4fb534-f1fe-4fd2-bc84-35369cddcd1c-kube-api-access-2nx7r\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-2c8f56-9b98j\" (UID: \"4c4fb534-f1fe-4fd2-bc84-35369cddcd1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2c8f56-9b98j" Oct 30 22:06:42.406163 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:42.406134 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2c8f56-9b98j" Oct 30 22:06:42.781880 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:42.781852 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2c8f56-9b98j"] Oct 30 22:06:42.786834 ip-10-0-143-61 kubenswrapper[5748]: W1030 22:06:42.786002 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c4fb534_f1fe_4fd2_bc84_35369cddcd1c.slice/crio-f598158ff028b7af6445bcd4dcd0ed1b1be32b19f66a0b233a56eb1b37a0e91b WatchSource:0}: Error finding container f598158ff028b7af6445bcd4dcd0ed1b1be32b19f66a0b233a56eb1b37a0e91b: Status 404 returned error can't find the container with id f598158ff028b7af6445bcd4dcd0ed1b1be32b19f66a0b233a56eb1b37a0e91b Oct 30 22:06:43.578694 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:43.578655 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2c8f56-9b98j" event={"ID":"4c4fb534-f1fe-4fd2-bc84-35369cddcd1c","Type":"ContainerStarted","Data":"57ff2698c6cf57752bd9a4ee0e7661daa9fa1535b469427538de031b8e27c1fa"} Oct 30 22:06:43.579119 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:43.578722 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2c8f56-9b98j" event={"ID":"4c4fb534-f1fe-4fd2-bc84-35369cddcd1c","Type":"ContainerStarted","Data":"f598158ff028b7af6445bcd4dcd0ed1b1be32b19f66a0b233a56eb1b37a0e91b"} Oct 30 22:06:44.585775 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:44.585740 5748 generic.go:356] "Generic (PLEG): container finished" podID="4c4fb534-f1fe-4fd2-bc84-35369cddcd1c" containerID="57ff2698c6cf57752bd9a4ee0e7661daa9fa1535b469427538de031b8e27c1fa" exitCode=0 Oct 30 22:06:44.586202 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:44.585821 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2c8f56-9b98j" event={"ID":"4c4fb534-f1fe-4fd2-bc84-35369cddcd1c","Type":"ContainerDied","Data":"57ff2698c6cf57752bd9a4ee0e7661daa9fa1535b469427538de031b8e27c1fa"} Oct 30 22:06:45.732693 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:45.732669 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2c8f56-9b98j" Oct 30 22:06:45.824959 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:45.824923 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c4fb534-f1fe-4fd2-bc84-35369cddcd1c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c4fb534-f1fe-4fd2-bc84-35369cddcd1c-maas-api-xks-ca\") pod \"4c4fb534-f1fe-4fd2-bc84-35369cddcd1c\" (UID: \"4c4fb534-f1fe-4fd2-bc84-35369cddcd1c\") " Oct 30 22:06:45.825162 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:45.825052 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c4fb534-f1fe-4fd2-bc84-35369cddcd1c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c4fb534-f1fe-4fd2-bc84-35369cddcd1c-maas-api-openshift-ca\") pod \"4c4fb534-f1fe-4fd2-bc84-35369cddcd1c\" (UID: \"4c4fb534-f1fe-4fd2-bc84-35369cddcd1c\") " Oct 30 22:06:45.825162 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:45.825161 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c4fb534-f1fe-4fd2-bc84-35369cddcd1c-kube-api-access-2nx7r\" (UniqueName: \"kubernetes.io/projected/4c4fb534-f1fe-4fd2-bc84-35369cddcd1c-kube-api-access-2nx7r\") pod \"4c4fb534-f1fe-4fd2-bc84-35369cddcd1c\" (UID: \"4c4fb534-f1fe-4fd2-bc84-35369cddcd1c\") " Oct 30 22:06:45.825403 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:45.825375 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c4fb534-f1fe-4fd2-bc84-35369cddcd1c-maas-api-openshift-ca" pod "4c4fb534-f1fe-4fd2-bc84-35369cddcd1c" (UID: "4c4fb534-f1fe-4fd2-bc84-35369cddcd1c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:06:45.825563 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:45.825546 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c4fb534-f1fe-4fd2-bc84-35369cddcd1c-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:06:45.827144 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:45.827117 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c4fb534-f1fe-4fd2-bc84-35369cddcd1c-maas-api-xks-ca" pod "4c4fb534-f1fe-4fd2-bc84-35369cddcd1c" (UID: "4c4fb534-f1fe-4fd2-bc84-35369cddcd1c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:06:45.827245 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:45.827212 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c4fb534-f1fe-4fd2-bc84-35369cddcd1c-kube-api-access-2nx7r" pod "4c4fb534-f1fe-4fd2-bc84-35369cddcd1c" (UID: "4c4fb534-f1fe-4fd2-bc84-35369cddcd1c"). InnerVolumeSpecName "kube-api-access-2nx7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:06:45.926775 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:45.926739 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nx7r\" (UniqueName: \"kubernetes.io/projected/4c4fb534-f1fe-4fd2-bc84-35369cddcd1c-kube-api-access-2nx7r\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:06:45.926775 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:45.926771 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c4fb534-f1fe-4fd2-bc84-35369cddcd1c-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:06:46.601192 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:46.601161 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2c8f56-9b98j" Oct 30 22:06:46.601353 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:46.601159 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2c8f56-9b98j" event={"ID":"4c4fb534-f1fe-4fd2-bc84-35369cddcd1c","Type":"ContainerDied","Data":"f598158ff028b7af6445bcd4dcd0ed1b1be32b19f66a0b233a56eb1b37a0e91b"} Oct 30 22:06:46.601353 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:06:46.601287 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f598158ff028b7af6445bcd4dcd0ed1b1be32b19f66a0b233a56eb1b37a0e91b" Oct 30 22:07:00.147930 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:07:00.147865 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:07:01.281558 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:01.281517 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d68156-gqhpp"] Oct 30 22:07:01.331196 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:01.331164 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d68156-gqhpp"] Oct 30 22:07:02.417440 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:02.417407 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59df8c0f-237e-4d38-9d2d-0982d5f44743" path="/var/lib/kubelet/pods/59df8c0f-237e-4d38-9d2d-0982d5f44743/volumes" Oct 30 22:07:08.138731 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:08.138697 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf"] Oct 30 22:07:08.139262 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:08.138954 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf" podUID="424d4fd4-debd-42b5-a2ab-09eefaaefa77" containerName="istio-proxy" containerID="cri-o://bb68636f461e65e0fd439fcfed1a14b521d2cb95d7de9394b7bf603061e3e566" gracePeriod=30 Oct 30 22:07:11.733704 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:11.733659 5748 patch_prober.go:29] interesting pod/e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:07:11.734118 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:11.733733 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf" podUID="424d4fd4-debd-42b5-a2ab-09eefaaefa77" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:07:13.399020 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:13.398997 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf" Oct 30 22:07:13.564376 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:13.564301 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/424d4fd4-debd-42b5-a2ab-09eefaaefa77-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/424d4fd4-debd-42b5-a2ab-09eefaaefa77-istio-data\") pod \"424d4fd4-debd-42b5-a2ab-09eefaaefa77\" (UID: \"424d4fd4-debd-42b5-a2ab-09eefaaefa77\") " Oct 30 22:07:13.564376 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:13.564354 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/424d4fd4-debd-42b5-a2ab-09eefaaefa77-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/424d4fd4-debd-42b5-a2ab-09eefaaefa77-istio-podinfo\") pod \"424d4fd4-debd-42b5-a2ab-09eefaaefa77\" (UID: \"424d4fd4-debd-42b5-a2ab-09eefaaefa77\") " Oct 30 22:07:13.564595 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:13.564442 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/424d4fd4-debd-42b5-a2ab-09eefaaefa77-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/424d4fd4-debd-42b5-a2ab-09eefaaefa77-istiod-ca-cert\") pod \"424d4fd4-debd-42b5-a2ab-09eefaaefa77\" (UID: \"424d4fd4-debd-42b5-a2ab-09eefaaefa77\") " Oct 30 22:07:13.564595 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:13.564462 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/424d4fd4-debd-42b5-a2ab-09eefaaefa77-istio-token\" (UniqueName: \"kubernetes.io/projected/424d4fd4-debd-42b5-a2ab-09eefaaefa77-istio-token\") pod \"424d4fd4-debd-42b5-a2ab-09eefaaefa77\" (UID: \"424d4fd4-debd-42b5-a2ab-09eefaaefa77\") " Oct 30 22:07:13.564595 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:13.564481 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/424d4fd4-debd-42b5-a2ab-09eefaaefa77-kube-api-access-jhxml\" (UniqueName: \"kubernetes.io/projected/424d4fd4-debd-42b5-a2ab-09eefaaefa77-kube-api-access-jhxml\") pod \"424d4fd4-debd-42b5-a2ab-09eefaaefa77\" (UID: \"424d4fd4-debd-42b5-a2ab-09eefaaefa77\") " Oct 30 22:07:13.564595 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:13.564517 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/424d4fd4-debd-42b5-a2ab-09eefaaefa77-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/424d4fd4-debd-42b5-a2ab-09eefaaefa77-workload-socket\") pod \"424d4fd4-debd-42b5-a2ab-09eefaaefa77\" (UID: \"424d4fd4-debd-42b5-a2ab-09eefaaefa77\") " Oct 30 22:07:13.564595 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:13.564590 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/424d4fd4-debd-42b5-a2ab-09eefaaefa77-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/424d4fd4-debd-42b5-a2ab-09eefaaefa77-istio-envoy\") pod \"424d4fd4-debd-42b5-a2ab-09eefaaefa77\" (UID: \"424d4fd4-debd-42b5-a2ab-09eefaaefa77\") " Oct 30 22:07:13.564870 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:13.564591 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/424d4fd4-debd-42b5-a2ab-09eefaaefa77-istio-data" pod "424d4fd4-debd-42b5-a2ab-09eefaaefa77" (UID: "424d4fd4-debd-42b5-a2ab-09eefaaefa77"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:07:13.564870 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:13.564667 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/424d4fd4-debd-42b5-a2ab-09eefaaefa77-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/424d4fd4-debd-42b5-a2ab-09eefaaefa77-workload-certs\") pod \"424d4fd4-debd-42b5-a2ab-09eefaaefa77\" (UID: \"424d4fd4-debd-42b5-a2ab-09eefaaefa77\") " Oct 30 22:07:13.564870 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:13.564702 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/424d4fd4-debd-42b5-a2ab-09eefaaefa77-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/424d4fd4-debd-42b5-a2ab-09eefaaefa77-credential-socket\") pod \"424d4fd4-debd-42b5-a2ab-09eefaaefa77\" (UID: \"424d4fd4-debd-42b5-a2ab-09eefaaefa77\") " Oct 30 22:07:13.564870 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:13.564856 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/424d4fd4-debd-42b5-a2ab-09eefaaefa77-workload-socket" pod "424d4fd4-debd-42b5-a2ab-09eefaaefa77" (UID: "424d4fd4-debd-42b5-a2ab-09eefaaefa77"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:07:13.565131 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:13.564950 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/424d4fd4-debd-42b5-a2ab-09eefaaefa77-istiod-ca-cert" pod "424d4fd4-debd-42b5-a2ab-09eefaaefa77" (UID: "424d4fd4-debd-42b5-a2ab-09eefaaefa77"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:07:13.565131 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:13.565087 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/424d4fd4-debd-42b5-a2ab-09eefaaefa77-credential-socket" pod "424d4fd4-debd-42b5-a2ab-09eefaaefa77" (UID: "424d4fd4-debd-42b5-a2ab-09eefaaefa77"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:07:13.565131 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:13.565100 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/424d4fd4-debd-42b5-a2ab-09eefaaefa77-workload-certs" pod "424d4fd4-debd-42b5-a2ab-09eefaaefa77" (UID: "424d4fd4-debd-42b5-a2ab-09eefaaefa77"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:07:13.565383 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:13.565362 5748 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/424d4fd4-debd-42b5-a2ab-09eefaaefa77-istiod-ca-cert\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:07:13.565451 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:13.565383 5748 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/424d4fd4-debd-42b5-a2ab-09eefaaefa77-workload-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:07:13.565451 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:13.565392 5748 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/424d4fd4-debd-42b5-a2ab-09eefaaefa77-workload-certs\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:07:13.565451 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:13.565403 5748 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/424d4fd4-debd-42b5-a2ab-09eefaaefa77-credential-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:07:13.565451 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:13.565417 5748 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/424d4fd4-debd-42b5-a2ab-09eefaaefa77-istio-data\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:07:13.566853 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:13.566821 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/424d4fd4-debd-42b5-a2ab-09eefaaefa77-istio-token" pod "424d4fd4-debd-42b5-a2ab-09eefaaefa77" (UID: "424d4fd4-debd-42b5-a2ab-09eefaaefa77"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:07:13.566853 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:13.566837 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/424d4fd4-debd-42b5-a2ab-09eefaaefa77-kube-api-access-jhxml" pod "424d4fd4-debd-42b5-a2ab-09eefaaefa77" (UID: "424d4fd4-debd-42b5-a2ab-09eefaaefa77"). InnerVolumeSpecName "kube-api-access-jhxml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:07:13.567064 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:13.566983 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/424d4fd4-debd-42b5-a2ab-09eefaaefa77-istio-podinfo" pod "424d4fd4-debd-42b5-a2ab-09eefaaefa77" (UID: "424d4fd4-debd-42b5-a2ab-09eefaaefa77"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:07:13.567206 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:13.567191 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/424d4fd4-debd-42b5-a2ab-09eefaaefa77-istio-envoy" pod "424d4fd4-debd-42b5-a2ab-09eefaaefa77" (UID: "424d4fd4-debd-42b5-a2ab-09eefaaefa77"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:07:13.667340 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:13.667306 5748 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/424d4fd4-debd-42b5-a2ab-09eefaaefa77-istio-podinfo\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:07:13.667340 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:13.667342 5748 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/424d4fd4-debd-42b5-a2ab-09eefaaefa77-istio-token\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:07:13.667532 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:13.667360 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhxml\" (UniqueName: \"kubernetes.io/projected/424d4fd4-debd-42b5-a2ab-09eefaaefa77-kube-api-access-jhxml\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:07:13.667532 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:13.667375 5748 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/424d4fd4-debd-42b5-a2ab-09eefaaefa77-istio-envoy\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:07:13.766443 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:13.766412 5748 generic.go:356] "Generic (PLEG): container finished" podID="424d4fd4-debd-42b5-a2ab-09eefaaefa77" containerID="bb68636f461e65e0fd439fcfed1a14b521d2cb95d7de9394b7bf603061e3e566" exitCode=0 Oct 30 22:07:13.766619 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:13.766456 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf" Oct 30 22:07:13.766619 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:13.766490 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf" event={"ID":"424d4fd4-debd-42b5-a2ab-09eefaaefa77","Type":"ContainerDied","Data":"bb68636f461e65e0fd439fcfed1a14b521d2cb95d7de9394b7bf603061e3e566"} Oct 30 22:07:13.766619 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:13.766542 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf" event={"ID":"424d4fd4-debd-42b5-a2ab-09eefaaefa77","Type":"ContainerDied","Data":"c4cdf1b6fc40bd4eb1c6dbee68f13409676a594b4c6c0ff8c54af20c2c046b10"} Oct 30 22:07:13.766619 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:13.766569 5748 scope.go:122] "RemoveContainer" containerID="bb68636f461e65e0fd439fcfed1a14b521d2cb95d7de9394b7bf603061e3e566" Oct 30 22:07:13.778319 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:13.778298 5748 scope.go:122] "RemoveContainer" containerID="bb68636f461e65e0fd439fcfed1a14b521d2cb95d7de9394b7bf603061e3e566" Oct 30 22:07:13.778608 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:07:13.778589 5748 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb68636f461e65e0fd439fcfed1a14b521d2cb95d7de9394b7bf603061e3e566\": container with ID starting with bb68636f461e65e0fd439fcfed1a14b521d2cb95d7de9394b7bf603061e3e566 not found: ID does not exist" containerID="bb68636f461e65e0fd439fcfed1a14b521d2cb95d7de9394b7bf603061e3e566" Oct 30 22:07:13.778685 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:13.778622 5748 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb68636f461e65e0fd439fcfed1a14b521d2cb95d7de9394b7bf603061e3e566"} err="failed to get container status \"bb68636f461e65e0fd439fcfed1a14b521d2cb95d7de9394b7bf603061e3e566\": rpc error: code = NotFound desc = could not find container \"bb68636f461e65e0fd439fcfed1a14b521d2cb95d7de9394b7bf603061e3e566\": container with ID starting with bb68636f461e65e0fd439fcfed1a14b521d2cb95d7de9394b7bf603061e3e566 not found: ID does not exist" Oct 30 22:07:13.934400 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:13.934375 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf"] Oct 30 22:07:13.940841 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:13.940810 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-t5ptf"] Oct 30 22:07:14.416761 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:14.416728 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="424d4fd4-debd-42b5-a2ab-09eefaaefa77" path="/var/lib/kubelet/pods/424d4fd4-debd-42b5-a2ab-09eefaaefa77/volumes" Oct 30 22:07:30.145865 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:07:30.145824 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:07:37.251009 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:37.250972 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2c8f56-9b98j"] Oct 30 22:07:37.301368 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:37.301336 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2c8f56-9b98j"] Oct 30 22:07:38.204536 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:38.204508 5748 scope.go:122] "RemoveContainer" containerID="6be82cad2f5e56dc0d2d5e022fa2b7b2de14ba4eabdf60c958b2aea6cd1aae8d" Oct 30 22:07:38.423850 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:38.423804 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c4fb534-f1fe-4fd2-bc84-35369cddcd1c" path="/var/lib/kubelet/pods/4c4fb534-f1fe-4fd2-bc84-35369cddcd1c/volumes" Oct 30 22:07:45.333727 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:45.333688 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-1a415f-x2js4"] Oct 30 22:07:45.334240 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:45.334226 5748 cpu_manager.go:414] "containerMap: removing container" podUID="424d4fd4-debd-42b5-a2ab-09eefaaefa77" containerName="istio-proxy" Oct 30 22:07:45.334293 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:45.334242 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="424d4fd4-debd-42b5-a2ab-09eefaaefa77" containerName="istio-proxy" Oct 30 22:07:45.334293 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:45.334254 5748 cpu_manager.go:414] "containerMap: removing container" podUID="4c4fb534-f1fe-4fd2-bc84-35369cddcd1c" containerName="revoke-keys" Oct 30 22:07:45.334293 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:45.334260 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c4fb534-f1fe-4fd2-bc84-35369cddcd1c" containerName="revoke-keys" Oct 30 22:07:45.334410 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:45.334345 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="424d4fd4-debd-42b5-a2ab-09eefaaefa77" containerName="istio-proxy" Oct 30 22:07:45.334410 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:45.334355 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c4fb534-f1fe-4fd2-bc84-35369cddcd1c" containerName="revoke-keys" Oct 30 22:07:45.339468 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:45.339445 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-1a415f-x2js4" Oct 30 22:07:45.393684 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:45.393657 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:07:45.393859 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:45.393762 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:07:45.394194 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:45.394156 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:07:45.394316 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:45.394217 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 22:07:45.396352 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:45.396330 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-1a415f-x2js4"] Oct 30 22:07:45.440030 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:45.440009 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4ce3691-6832-44ca-a3bf-34205dd697b6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-1a415f-x2js4\" (UID: \"a4ce3691-6832-44ca-a3bf-34205dd697b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-1a415f-x2js4" Oct 30 22:07:45.440211 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:45.440193 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p82q\" (UniqueName: \"kubernetes.io/projected/a4ce3691-6832-44ca-a3bf-34205dd697b6-kube-api-access-5p82q\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-1a415f-x2js4\" (UID: \"a4ce3691-6832-44ca-a3bf-34205dd697b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-1a415f-x2js4" Oct 30 22:07:45.440266 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:45.440231 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4ce3691-6832-44ca-a3bf-34205dd697b6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-1a415f-x2js4\" (UID: \"a4ce3691-6832-44ca-a3bf-34205dd697b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-1a415f-x2js4" Oct 30 22:07:45.541736 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:45.541713 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p82q\" (UniqueName: \"kubernetes.io/projected/a4ce3691-6832-44ca-a3bf-34205dd697b6-kube-api-access-5p82q\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-1a415f-x2js4\" (UID: \"a4ce3691-6832-44ca-a3bf-34205dd697b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-1a415f-x2js4" Oct 30 22:07:45.541861 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:45.541750 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4ce3691-6832-44ca-a3bf-34205dd697b6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-1a415f-x2js4\" (UID: \"a4ce3691-6832-44ca-a3bf-34205dd697b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-1a415f-x2js4" Oct 30 22:07:45.541861 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:45.541780 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4ce3691-6832-44ca-a3bf-34205dd697b6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-1a415f-x2js4\" (UID: \"a4ce3691-6832-44ca-a3bf-34205dd697b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-1a415f-x2js4" Oct 30 22:07:45.542478 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:45.542455 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4ce3691-6832-44ca-a3bf-34205dd697b6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-1a415f-x2js4\" (UID: \"a4ce3691-6832-44ca-a3bf-34205dd697b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-1a415f-x2js4" Oct 30 22:07:45.544018 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:45.543994 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4ce3691-6832-44ca-a3bf-34205dd697b6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-1a415f-x2js4\" (UID: \"a4ce3691-6832-44ca-a3bf-34205dd697b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-1a415f-x2js4" Oct 30 22:07:45.605190 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:45.605138 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p82q\" (UniqueName: \"kubernetes.io/projected/a4ce3691-6832-44ca-a3bf-34205dd697b6-kube-api-access-5p82q\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-1a415f-x2js4\" (UID: \"a4ce3691-6832-44ca-a3bf-34205dd697b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-1a415f-x2js4" Oct 30 22:07:45.653173 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:45.653146 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-1a415f-x2js4" Oct 30 22:07:46.030930 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:46.030891 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-1a415f-x2js4"] Oct 30 22:07:46.033312 ip-10-0-143-61 kubenswrapper[5748]: W1030 22:07:46.033281 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4ce3691_6832_44ca_a3bf_34205dd697b6.slice/crio-89f5ea709469f11187634583c5f8351776c14c911bad43c53bd3fafed1898523 WatchSource:0}: Error finding container 89f5ea709469f11187634583c5f8351776c14c911bad43c53bd3fafed1898523: Status 404 returned error can't find the container with id 89f5ea709469f11187634583c5f8351776c14c911bad43c53bd3fafed1898523 Oct 30 22:07:46.050147 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:46.050124 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f487998b-98lsz"] Oct 30 22:07:46.054159 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:46.054138 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f80439bf-drncz"] Oct 30 22:07:46.081728 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:46.081706 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f487998b-98lsz"] Oct 30 22:07:46.083729 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:46.083707 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f80439bf-drncz"] Oct 30 22:07:46.417350 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:46.417317 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="353fc156-d3cc-4d53-bef6-581488884e82" path="/var/lib/kubelet/pods/353fc156-d3cc-4d53-bef6-581488884e82/volumes" Oct 30 22:07:46.417768 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:46.417753 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a48970bd-1994-40cd-a7e1-e731848c7da1" path="/var/lib/kubelet/pods/a48970bd-1994-40cd-a7e1-e731848c7da1/volumes" Oct 30 22:07:46.976530 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:46.976496 5748 generic.go:356] "Generic (PLEG): container finished" podID="a4ce3691-6832-44ca-a3bf-34205dd697b6" containerID="ce5ce3db0dfe5d7c7f3704a4e4bcd8927fc26c755b3f07307ee296112800f1b8" exitCode=0 Oct 30 22:07:46.976695 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:46.976584 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-1a415f-x2js4" event={"ID":"a4ce3691-6832-44ca-a3bf-34205dd697b6","Type":"ContainerDied","Data":"ce5ce3db0dfe5d7c7f3704a4e4bcd8927fc26c755b3f07307ee296112800f1b8"} Oct 30 22:07:46.976695 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:46.976647 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-1a415f-x2js4" event={"ID":"a4ce3691-6832-44ca-a3bf-34205dd697b6","Type":"ContainerStarted","Data":"89f5ea709469f11187634583c5f8351776c14c911bad43c53bd3fafed1898523"} Oct 30 22:07:48.127757 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:48.127732 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-1a415f-x2js4" Oct 30 22:07:48.274572 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:48.274486 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4ce3691-6832-44ca-a3bf-34205dd697b6-kube-api-access-5p82q\" (UniqueName: \"kubernetes.io/projected/a4ce3691-6832-44ca-a3bf-34205dd697b6-kube-api-access-5p82q\") pod \"a4ce3691-6832-44ca-a3bf-34205dd697b6\" (UID: \"a4ce3691-6832-44ca-a3bf-34205dd697b6\") " Oct 30 22:07:48.274729 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:48.274659 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4ce3691-6832-44ca-a3bf-34205dd697b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4ce3691-6832-44ca-a3bf-34205dd697b6-maas-api-openshift-ca\") pod \"a4ce3691-6832-44ca-a3bf-34205dd697b6\" (UID: \"a4ce3691-6832-44ca-a3bf-34205dd697b6\") " Oct 30 22:07:48.274801 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:48.274731 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4ce3691-6832-44ca-a3bf-34205dd697b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4ce3691-6832-44ca-a3bf-34205dd697b6-maas-api-xks-ca\") pod \"a4ce3691-6832-44ca-a3bf-34205dd697b6\" (UID: \"a4ce3691-6832-44ca-a3bf-34205dd697b6\") " Oct 30 22:07:48.275069 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:48.275037 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4ce3691-6832-44ca-a3bf-34205dd697b6-maas-api-openshift-ca" pod "a4ce3691-6832-44ca-a3bf-34205dd697b6" (UID: "a4ce3691-6832-44ca-a3bf-34205dd697b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:07:48.275240 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:48.275168 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4ce3691-6832-44ca-a3bf-34205dd697b6-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:07:48.276853 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:48.276829 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4ce3691-6832-44ca-a3bf-34205dd697b6-maas-api-xks-ca" pod "a4ce3691-6832-44ca-a3bf-34205dd697b6" (UID: "a4ce3691-6832-44ca-a3bf-34205dd697b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:07:48.276980 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:48.276958 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4ce3691-6832-44ca-a3bf-34205dd697b6-kube-api-access-5p82q" pod "a4ce3691-6832-44ca-a3bf-34205dd697b6" (UID: "a4ce3691-6832-44ca-a3bf-34205dd697b6"). InnerVolumeSpecName "kube-api-access-5p82q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:07:48.375989 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:48.375966 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4ce3691-6832-44ca-a3bf-34205dd697b6-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:07:48.375989 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:48.375989 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p82q\" (UniqueName: \"kubernetes.io/projected/a4ce3691-6832-44ca-a3bf-34205dd697b6-kube-api-access-5p82q\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:07:48.990966 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:48.990614 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-1a415f-x2js4" event={"ID":"a4ce3691-6832-44ca-a3bf-34205dd697b6","Type":"ContainerDied","Data":"89f5ea709469f11187634583c5f8351776c14c911bad43c53bd3fafed1898523"} Oct 30 22:07:48.991169 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:48.991029 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89f5ea709469f11187634583c5f8351776c14c911bad43c53bd3fafed1898523" Oct 30 22:07:48.991169 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:07:48.991148 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-1a415f-x2js4" Oct 30 22:08:00.145998 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:08:00.145959 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:08:28.459417 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:28.459371 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-98d38f-kh4cd"] Oct 30 22:08:28.508959 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:28.508925 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-98d38f-kh4cd"] Oct 30 22:08:29.875398 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:29.875366 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ae1bd5-h9xff"] Oct 30 22:08:29.925255 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:29.925224 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ae1bd5-h9xff"] Oct 30 22:08:30.144582 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:08:30.144543 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:08:30.417134 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:30.417060 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06e13f56-0557-4d59-b085-5a195344e67e" path="/var/lib/kubelet/pods/06e13f56-0557-4d59-b085-5a195344e67e/volumes" Oct 30 22:08:30.417505 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:30.417492 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="237d15fa-293c-4601-aa9e-3f233c758428" path="/var/lib/kubelet/pods/237d15fa-293c-4601-aa9e-3f233c758428/volumes" Oct 30 22:08:31.231638 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:31.231606 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4"] Oct 30 22:08:31.232073 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:31.231804 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4" podUID="2d063b03-bc00-48bf-a91c-dfb31c286788" containerName="istio-proxy" containerID="cri-o://3fb2a39df3ce04727343738955ed32b250dd0c4613daa99fee335f2bf00ccc25" gracePeriod=30 Oct 30 22:08:34.507081 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:34.507050 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtdx4"] Oct 30 22:08:34.507575 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:34.507230 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtdx4" podUID="f73607c3-58cc-480e-ba9d-08893ee92244" containerName="discovery" containerID="cri-o://1ad4f8c4ffeeed83d926fe082f25359a4ce8f0bec2fd821ca6c052b72394cd03" gracePeriod=30 Oct 30 22:08:34.781503 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:34.781475 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtdx4" Oct 30 22:08:34.822061 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:34.822023 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f73607c3-58cc-480e-ba9d-08893ee92244-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f73607c3-58cc-480e-ba9d-08893ee92244-local-certs\") pod \"f73607c3-58cc-480e-ba9d-08893ee92244\" (UID: \"f73607c3-58cc-480e-ba9d-08893ee92244\") " Oct 30 22:08:34.822277 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:34.822110 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f73607c3-58cc-480e-ba9d-08893ee92244-kube-api-access-clmb4\" (UniqueName: \"kubernetes.io/projected/f73607c3-58cc-480e-ba9d-08893ee92244-kube-api-access-clmb4\") pod \"f73607c3-58cc-480e-ba9d-08893ee92244\" (UID: \"f73607c3-58cc-480e-ba9d-08893ee92244\") " Oct 30 22:08:34.822277 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:34.822168 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f73607c3-58cc-480e-ba9d-08893ee92244-cacerts\" (UniqueName: \"kubernetes.io/secret/f73607c3-58cc-480e-ba9d-08893ee92244-cacerts\") pod \"f73607c3-58cc-480e-ba9d-08893ee92244\" (UID: \"f73607c3-58cc-480e-ba9d-08893ee92244\") " Oct 30 22:08:34.822277 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:34.822206 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f73607c3-58cc-480e-ba9d-08893ee92244-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f73607c3-58cc-480e-ba9d-08893ee92244-istio-csr-dns-cert\") pod \"f73607c3-58cc-480e-ba9d-08893ee92244\" (UID: \"f73607c3-58cc-480e-ba9d-08893ee92244\") " Oct 30 22:08:34.822277 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:34.822250 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f73607c3-58cc-480e-ba9d-08893ee92244-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f73607c3-58cc-480e-ba9d-08893ee92244-istio-kubeconfig\") pod \"f73607c3-58cc-480e-ba9d-08893ee92244\" (UID: \"f73607c3-58cc-480e-ba9d-08893ee92244\") " Oct 30 22:08:34.822512 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:34.822304 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f73607c3-58cc-480e-ba9d-08893ee92244-istio-token\" (UniqueName: \"kubernetes.io/projected/f73607c3-58cc-480e-ba9d-08893ee92244-istio-token\") pod \"f73607c3-58cc-480e-ba9d-08893ee92244\" (UID: \"f73607c3-58cc-480e-ba9d-08893ee92244\") " Oct 30 22:08:34.822512 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:34.822430 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f73607c3-58cc-480e-ba9d-08893ee92244-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f73607c3-58cc-480e-ba9d-08893ee92244-istio-csr-ca-configmap\") pod \"f73607c3-58cc-480e-ba9d-08893ee92244\" (UID: \"f73607c3-58cc-480e-ba9d-08893ee92244\") " Oct 30 22:08:34.823091 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:34.823049 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f73607c3-58cc-480e-ba9d-08893ee92244-istio-csr-ca-configmap" pod "f73607c3-58cc-480e-ba9d-08893ee92244" (UID: "f73607c3-58cc-480e-ba9d-08893ee92244"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:08:34.825565 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:34.825535 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f73607c3-58cc-480e-ba9d-08893ee92244-local-certs" pod "f73607c3-58cc-480e-ba9d-08893ee92244" (UID: "f73607c3-58cc-480e-ba9d-08893ee92244"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:08:34.825936 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:34.825912 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f73607c3-58cc-480e-ba9d-08893ee92244-istio-kubeconfig" pod "f73607c3-58cc-480e-ba9d-08893ee92244" (UID: "f73607c3-58cc-480e-ba9d-08893ee92244"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:08:34.826204 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:34.826172 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f73607c3-58cc-480e-ba9d-08893ee92244-cacerts" pod "f73607c3-58cc-480e-ba9d-08893ee92244" (UID: "f73607c3-58cc-480e-ba9d-08893ee92244"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:08:34.826326 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:34.826307 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f73607c3-58cc-480e-ba9d-08893ee92244-kube-api-access-clmb4" pod "f73607c3-58cc-480e-ba9d-08893ee92244" (UID: "f73607c3-58cc-480e-ba9d-08893ee92244"). InnerVolumeSpecName "kube-api-access-clmb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:34.826385 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:34.826355 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f73607c3-58cc-480e-ba9d-08893ee92244-istio-token" pod "f73607c3-58cc-480e-ba9d-08893ee92244" (UID: "f73607c3-58cc-480e-ba9d-08893ee92244"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:34.826614 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:34.826592 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f73607c3-58cc-480e-ba9d-08893ee92244-istio-csr-dns-cert" pod "f73607c3-58cc-480e-ba9d-08893ee92244" (UID: "f73607c3-58cc-480e-ba9d-08893ee92244"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:08:34.924006 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:34.923970 5748 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f73607c3-58cc-480e-ba9d-08893ee92244-local-certs\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:34.924165 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:34.924010 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clmb4\" (UniqueName: \"kubernetes.io/projected/f73607c3-58cc-480e-ba9d-08893ee92244-kube-api-access-clmb4\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:34.924165 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:34.924027 5748 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f73607c3-58cc-480e-ba9d-08893ee92244-cacerts\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:34.924165 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:34.924041 5748 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f73607c3-58cc-480e-ba9d-08893ee92244-istio-csr-dns-cert\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:34.924165 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:34.924054 5748 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f73607c3-58cc-480e-ba9d-08893ee92244-istio-kubeconfig\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:34.924165 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:34.924068 5748 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f73607c3-58cc-480e-ba9d-08893ee92244-istio-token\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:34.924165 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:34.924082 5748 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f73607c3-58cc-480e-ba9d-08893ee92244-istio-csr-ca-configmap\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:35.313211 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:35.313178 5748 generic.go:356] "Generic (PLEG): container finished" podID="f73607c3-58cc-480e-ba9d-08893ee92244" containerID="1ad4f8c4ffeeed83d926fe082f25359a4ce8f0bec2fd821ca6c052b72394cd03" exitCode=0 Oct 30 22:08:35.313402 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:35.313230 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtdx4" Oct 30 22:08:35.313402 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:35.313266 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtdx4" event={"ID":"f73607c3-58cc-480e-ba9d-08893ee92244","Type":"ContainerDied","Data":"1ad4f8c4ffeeed83d926fe082f25359a4ce8f0bec2fd821ca6c052b72394cd03"} Oct 30 22:08:35.313402 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:35.313321 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtdx4" event={"ID":"f73607c3-58cc-480e-ba9d-08893ee92244","Type":"ContainerDied","Data":"bf03a221b7ae9e84d17ff1d90832ab10af6b56d5068a2b46b325452cf9878e89"} Oct 30 22:08:35.313402 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:35.313343 5748 scope.go:122] "RemoveContainer" containerID="1ad4f8c4ffeeed83d926fe082f25359a4ce8f0bec2fd821ca6c052b72394cd03" Oct 30 22:08:35.325060 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:35.325035 5748 scope.go:122] "RemoveContainer" containerID="1ad4f8c4ffeeed83d926fe082f25359a4ce8f0bec2fd821ca6c052b72394cd03" Oct 30 22:08:35.325332 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:08:35.325314 5748 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ad4f8c4ffeeed83d926fe082f25359a4ce8f0bec2fd821ca6c052b72394cd03\": container with ID starting with 1ad4f8c4ffeeed83d926fe082f25359a4ce8f0bec2fd821ca6c052b72394cd03 not found: ID does not exist" containerID="1ad4f8c4ffeeed83d926fe082f25359a4ce8f0bec2fd821ca6c052b72394cd03" Oct 30 22:08:35.325379 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:35.325341 5748 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ad4f8c4ffeeed83d926fe082f25359a4ce8f0bec2fd821ca6c052b72394cd03"} err="failed to get container status \"1ad4f8c4ffeeed83d926fe082f25359a4ce8f0bec2fd821ca6c052b72394cd03\": rpc error: code = NotFound desc = could not find container \"1ad4f8c4ffeeed83d926fe082f25359a4ce8f0bec2fd821ca6c052b72394cd03\": container with ID starting with 1ad4f8c4ffeeed83d926fe082f25359a4ce8f0bec2fd821ca6c052b72394cd03 not found: ID does not exist" Oct 30 22:08:35.483743 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:35.483712 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtdx4"] Oct 30 22:08:35.534167 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:35.534136 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtdx4"] Oct 30 22:08:36.422212 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:36.422180 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f73607c3-58cc-480e-ba9d-08893ee92244" path="/var/lib/kubelet/pods/f73607c3-58cc-480e-ba9d-08893ee92244/volumes" Oct 30 22:08:36.547384 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:36.547355 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4" Oct 30 22:08:36.649006 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:36.648967 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d063b03-bc00-48bf-a91c-dfb31c286788-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d063b03-bc00-48bf-a91c-dfb31c286788-workload-socket\") pod \"2d063b03-bc00-48bf-a91c-dfb31c286788\" (UID: \"2d063b03-bc00-48bf-a91c-dfb31c286788\") " Oct 30 22:08:36.649223 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:36.649034 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d063b03-bc00-48bf-a91c-dfb31c286788-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d063b03-bc00-48bf-a91c-dfb31c286788-istiod-ca-cert\") pod \"2d063b03-bc00-48bf-a91c-dfb31c286788\" (UID: \"2d063b03-bc00-48bf-a91c-dfb31c286788\") " Oct 30 22:08:36.649223 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:36.649113 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d063b03-bc00-48bf-a91c-dfb31c286788-kube-api-access-cfw65\" (UniqueName: \"kubernetes.io/projected/2d063b03-bc00-48bf-a91c-dfb31c286788-kube-api-access-cfw65\") pod \"2d063b03-bc00-48bf-a91c-dfb31c286788\" (UID: \"2d063b03-bc00-48bf-a91c-dfb31c286788\") " Oct 30 22:08:36.649223 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:36.649168 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d063b03-bc00-48bf-a91c-dfb31c286788-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d063b03-bc00-48bf-a91c-dfb31c286788-istio-data\") pod \"2d063b03-bc00-48bf-a91c-dfb31c286788\" (UID: \"2d063b03-bc00-48bf-a91c-dfb31c286788\") " Oct 30 22:08:36.649223 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:36.649208 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d063b03-bc00-48bf-a91c-dfb31c286788-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d063b03-bc00-48bf-a91c-dfb31c286788-istio-podinfo\") pod \"2d063b03-bc00-48bf-a91c-dfb31c286788\" (UID: \"2d063b03-bc00-48bf-a91c-dfb31c286788\") " Oct 30 22:08:36.649545 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:36.649270 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d063b03-bc00-48bf-a91c-dfb31c286788-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d063b03-bc00-48bf-a91c-dfb31c286788-istio-envoy\") pod \"2d063b03-bc00-48bf-a91c-dfb31c286788\" (UID: \"2d063b03-bc00-48bf-a91c-dfb31c286788\") " Oct 30 22:08:36.649545 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:36.649281 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d063b03-bc00-48bf-a91c-dfb31c286788-workload-socket" pod "2d063b03-bc00-48bf-a91c-dfb31c286788" (UID: "2d063b03-bc00-48bf-a91c-dfb31c286788"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:08:36.649545 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:36.649341 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d063b03-bc00-48bf-a91c-dfb31c286788-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d063b03-bc00-48bf-a91c-dfb31c286788-credential-socket\") pod \"2d063b03-bc00-48bf-a91c-dfb31c286788\" (UID: \"2d063b03-bc00-48bf-a91c-dfb31c286788\") " Oct 30 22:08:36.650225 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:36.649972 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d063b03-bc00-48bf-a91c-dfb31c286788-istiod-ca-cert" pod "2d063b03-bc00-48bf-a91c-dfb31c286788" (UID: "2d063b03-bc00-48bf-a91c-dfb31c286788"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:08:36.650471 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:36.650436 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d063b03-bc00-48bf-a91c-dfb31c286788-istio-data" pod "2d063b03-bc00-48bf-a91c-dfb31c286788" (UID: "2d063b03-bc00-48bf-a91c-dfb31c286788"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:08:36.650598 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:36.650565 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d063b03-bc00-48bf-a91c-dfb31c286788-credential-socket" pod "2d063b03-bc00-48bf-a91c-dfb31c286788" (UID: "2d063b03-bc00-48bf-a91c-dfb31c286788"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:08:36.650730 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:36.650713 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d063b03-bc00-48bf-a91c-dfb31c286788-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d063b03-bc00-48bf-a91c-dfb31c286788-workload-certs\") pod \"2d063b03-bc00-48bf-a91c-dfb31c286788\" (UID: \"2d063b03-bc00-48bf-a91c-dfb31c286788\") " Oct 30 22:08:36.651280 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:36.651103 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d063b03-bc00-48bf-a91c-dfb31c286788-istio-token\" (UniqueName: \"kubernetes.io/projected/2d063b03-bc00-48bf-a91c-dfb31c286788-istio-token\") pod \"2d063b03-bc00-48bf-a91c-dfb31c286788\" (UID: \"2d063b03-bc00-48bf-a91c-dfb31c286788\") " Oct 30 22:08:36.651855 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:36.651040 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d063b03-bc00-48bf-a91c-dfb31c286788-workload-certs" pod "2d063b03-bc00-48bf-a91c-dfb31c286788" (UID: "2d063b03-bc00-48bf-a91c-dfb31c286788"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:08:36.652726 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:36.652708 5748 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d063b03-bc00-48bf-a91c-dfb31c286788-workload-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:36.652936 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:36.652919 5748 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d063b03-bc00-48bf-a91c-dfb31c286788-istiod-ca-cert\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:36.654054 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:36.654035 5748 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d063b03-bc00-48bf-a91c-dfb31c286788-istio-data\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:36.654180 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:36.654165 5748 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d063b03-bc00-48bf-a91c-dfb31c286788-credential-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:36.654286 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:36.654273 5748 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d063b03-bc00-48bf-a91c-dfb31c286788-workload-certs\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:36.656481 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:36.656452 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d063b03-bc00-48bf-a91c-dfb31c286788-kube-api-access-cfw65" pod "2d063b03-bc00-48bf-a91c-dfb31c286788" (UID: "2d063b03-bc00-48bf-a91c-dfb31c286788"). InnerVolumeSpecName "kube-api-access-cfw65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:36.657398 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:36.657348 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d063b03-bc00-48bf-a91c-dfb31c286788-istio-token" pod "2d063b03-bc00-48bf-a91c-dfb31c286788" (UID: "2d063b03-bc00-48bf-a91c-dfb31c286788"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:36.659582 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:36.659536 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d063b03-bc00-48bf-a91c-dfb31c286788-istio-podinfo" pod "2d063b03-bc00-48bf-a91c-dfb31c286788" (UID: "2d063b03-bc00-48bf-a91c-dfb31c286788"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:08:36.660816 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:36.660785 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d063b03-bc00-48bf-a91c-dfb31c286788-istio-envoy" pod "2d063b03-bc00-48bf-a91c-dfb31c286788" (UID: "2d063b03-bc00-48bf-a91c-dfb31c286788"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:08:36.755720 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:36.755642 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfw65\" (UniqueName: \"kubernetes.io/projected/2d063b03-bc00-48bf-a91c-dfb31c286788-kube-api-access-cfw65\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:36.755720 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:36.755679 5748 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d063b03-bc00-48bf-a91c-dfb31c286788-istio-podinfo\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:36.755720 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:36.755695 5748 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d063b03-bc00-48bf-a91c-dfb31c286788-istio-envoy\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:36.755720 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:36.755709 5748 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d063b03-bc00-48bf-a91c-dfb31c286788-istio-token\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:36.872406 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:36.872368 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j4v8r_3df036ff-1c00-4ce2-9e86-8acbeedf6c82/ovn-acl-logging/0.log" Oct 30 22:08:36.877171 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:36.877150 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j4v8r_3df036ff-1c00-4ce2-9e86-8acbeedf6c82/ovn-acl-logging/0.log" Oct 30 22:08:37.328190 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:37.328156 5748 generic.go:356] "Generic (PLEG): container finished" podID="2d063b03-bc00-48bf-a91c-dfb31c286788" containerID="3fb2a39df3ce04727343738955ed32b250dd0c4613daa99fee335f2bf00ccc25" exitCode=0 Oct 30 22:08:37.328367 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:37.328209 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4" Oct 30 22:08:37.328367 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:37.328246 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4" event={"ID":"2d063b03-bc00-48bf-a91c-dfb31c286788","Type":"ContainerDied","Data":"3fb2a39df3ce04727343738955ed32b250dd0c4613daa99fee335f2bf00ccc25"} Oct 30 22:08:37.328367 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:37.328321 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4" event={"ID":"2d063b03-bc00-48bf-a91c-dfb31c286788","Type":"ContainerDied","Data":"d7cabf8a695a2b17f2629a620802f08a729b21ee9303e6a135e690885a5aebd8"} Oct 30 22:08:37.328367 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:37.328343 5748 scope.go:122] "RemoveContainer" containerID="3fb2a39df3ce04727343738955ed32b250dd0c4613daa99fee335f2bf00ccc25" Oct 30 22:08:37.338944 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:37.338925 5748 scope.go:122] "RemoveContainer" containerID="3fb2a39df3ce04727343738955ed32b250dd0c4613daa99fee335f2bf00ccc25" Oct 30 22:08:37.339210 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:08:37.339189 5748 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fb2a39df3ce04727343738955ed32b250dd0c4613daa99fee335f2bf00ccc25\": container with ID starting with 3fb2a39df3ce04727343738955ed32b250dd0c4613daa99fee335f2bf00ccc25 not found: ID does not exist" containerID="3fb2a39df3ce04727343738955ed32b250dd0c4613daa99fee335f2bf00ccc25" Oct 30 22:08:37.339262 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:37.339217 5748 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fb2a39df3ce04727343738955ed32b250dd0c4613daa99fee335f2bf00ccc25"} err="failed to get container status \"3fb2a39df3ce04727343738955ed32b250dd0c4613daa99fee335f2bf00ccc25\": rpc error: code = NotFound desc = could not find container \"3fb2a39df3ce04727343738955ed32b250dd0c4613daa99fee335f2bf00ccc25\": container with ID starting with 3fb2a39df3ce04727343738955ed32b250dd0c4613daa99fee335f2bf00ccc25 not found: ID does not exist" Oct 30 22:08:37.361795 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:37.361775 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5bf4f8-ptnj9"] Oct 30 22:08:37.362279 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:37.362253 5748 cpu_manager.go:414] "containerMap: removing container" podUID="2d063b03-bc00-48bf-a91c-dfb31c286788" containerName="istio-proxy" Oct 30 22:08:37.362279 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:37.362281 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d063b03-bc00-48bf-a91c-dfb31c286788" containerName="istio-proxy" Oct 30 22:08:37.362366 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:37.362300 5748 cpu_manager.go:414] "containerMap: removing container" podUID="a4ce3691-6832-44ca-a3bf-34205dd697b6" containerName="revoke-keys" Oct 30 22:08:37.362366 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:37.362305 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4ce3691-6832-44ca-a3bf-34205dd697b6" containerName="revoke-keys" Oct 30 22:08:37.362366 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:37.362327 5748 cpu_manager.go:414] "containerMap: removing container" podUID="f73607c3-58cc-480e-ba9d-08893ee92244" containerName="discovery" Oct 30 22:08:37.362366 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:37.362332 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f73607c3-58cc-480e-ba9d-08893ee92244" containerName="discovery" Oct 30 22:08:37.362549 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:37.362418 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="f73607c3-58cc-480e-ba9d-08893ee92244" containerName="discovery" Oct 30 22:08:37.362549 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:37.362433 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d063b03-bc00-48bf-a91c-dfb31c286788" containerName="istio-proxy" Oct 30 22:08:37.362549 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:37.362440 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4ce3691-6832-44ca-a3bf-34205dd697b6" containerName="revoke-keys" Oct 30 22:08:37.366939 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:37.366921 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5bf4f8-ptnj9" Oct 30 22:08:37.431912 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:37.431865 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:08:37.432066 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:37.431919 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:08:37.432066 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:37.431952 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:08:37.432066 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:37.432040 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 22:08:37.434871 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:37.434849 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5bf4f8-ptnj9"] Oct 30 22:08:37.463118 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:37.463087 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ee699aa-aa04-4b9f-8f92-4d1c33a15034-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5bf4f8-ptnj9\" (UID: \"2ee699aa-aa04-4b9f-8f92-4d1c33a15034\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5bf4f8-ptnj9" Oct 30 22:08:37.463252 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:37.463134 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qknm7\" (UniqueName: \"kubernetes.io/projected/2ee699aa-aa04-4b9f-8f92-4d1c33a15034-kube-api-access-qknm7\") pod \"maas-api-revoke-keys-e2e-api-b-5bf4f8-ptnj9\" (UID: \"2ee699aa-aa04-4b9f-8f92-4d1c33a15034\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5bf4f8-ptnj9" Oct 30 22:08:37.463481 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:37.463452 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ee699aa-aa04-4b9f-8f92-4d1c33a15034-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5bf4f8-ptnj9\" (UID: \"2ee699aa-aa04-4b9f-8f92-4d1c33a15034\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5bf4f8-ptnj9" Oct 30 22:08:37.524598 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:37.524569 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4"] Oct 30 22:08:37.529283 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:37.529259 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-kt2x4"] Oct 30 22:08:37.564666 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:37.564641 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ee699aa-aa04-4b9f-8f92-4d1c33a15034-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5bf4f8-ptnj9\" (UID: \"2ee699aa-aa04-4b9f-8f92-4d1c33a15034\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5bf4f8-ptnj9" Oct 30 22:08:37.565108 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:37.564673 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qknm7\" (UniqueName: \"kubernetes.io/projected/2ee699aa-aa04-4b9f-8f92-4d1c33a15034-kube-api-access-qknm7\") pod \"maas-api-revoke-keys-e2e-api-b-5bf4f8-ptnj9\" (UID: \"2ee699aa-aa04-4b9f-8f92-4d1c33a15034\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5bf4f8-ptnj9" Oct 30 22:08:37.565108 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:37.564742 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ee699aa-aa04-4b9f-8f92-4d1c33a15034-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5bf4f8-ptnj9\" (UID: \"2ee699aa-aa04-4b9f-8f92-4d1c33a15034\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5bf4f8-ptnj9" Oct 30 22:08:37.565408 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:37.565391 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ee699aa-aa04-4b9f-8f92-4d1c33a15034-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5bf4f8-ptnj9\" (UID: \"2ee699aa-aa04-4b9f-8f92-4d1c33a15034\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5bf4f8-ptnj9" Oct 30 22:08:37.567108 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:37.567085 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ee699aa-aa04-4b9f-8f92-4d1c33a15034-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5bf4f8-ptnj9\" (UID: \"2ee699aa-aa04-4b9f-8f92-4d1c33a15034\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5bf4f8-ptnj9" Oct 30 22:08:37.626511 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:37.626453 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qknm7\" (UniqueName: \"kubernetes.io/projected/2ee699aa-aa04-4b9f-8f92-4d1c33a15034-kube-api-access-qknm7\") pod \"maas-api-revoke-keys-e2e-api-b-5bf4f8-ptnj9\" (UID: \"2ee699aa-aa04-4b9f-8f92-4d1c33a15034\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5bf4f8-ptnj9" Oct 30 22:08:37.680196 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:37.680174 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5bf4f8-ptnj9" Oct 30 22:08:38.068357 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:38.068333 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5bf4f8-ptnj9"] Oct 30 22:08:38.069935 ip-10-0-143-61 kubenswrapper[5748]: W1030 22:08:38.069892 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ee699aa_aa04_4b9f_8f92_4d1c33a15034.slice/crio-6461fbbd6eb7e447115c8bea73377cd0d16826d1396fd4002b4539967ce6ea4a WatchSource:0}: Error finding container 6461fbbd6eb7e447115c8bea73377cd0d16826d1396fd4002b4539967ce6ea4a: Status 404 returned error can't find the container with id 6461fbbd6eb7e447115c8bea73377cd0d16826d1396fd4002b4539967ce6ea4a Oct 30 22:08:38.232716 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:38.232684 5748 scope.go:122] "RemoveContainer" containerID="815a6e6beab44301d0edcb0d11cae079e9a2f80884c78d70187fb7eac9390da3" Oct 30 22:08:38.335844 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:38.335781 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5bf4f8-ptnj9" event={"ID":"2ee699aa-aa04-4b9f-8f92-4d1c33a15034","Type":"ContainerStarted","Data":"6461fbbd6eb7e447115c8bea73377cd0d16826d1396fd4002b4539967ce6ea4a"} Oct 30 22:08:38.416637 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:38.416610 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d063b03-bc00-48bf-a91c-dfb31c286788" path="/var/lib/kubelet/pods/2d063b03-bc00-48bf-a91c-dfb31c286788/volumes" Oct 30 22:08:38.856793 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:38.856757 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5a19c1-mkwqn"] Oct 30 22:08:38.860721 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:38.860705 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5a19c1-mkwqn" Oct 30 22:08:38.919589 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:38.919547 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5a19c1-mkwqn"] Oct 30 22:08:38.981359 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:38.981322 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f344b29-02c0-40ec-96b3-3543ebda4d28-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5a19c1-mkwqn\" (UID: \"3f344b29-02c0-40ec-96b3-3543ebda4d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5a19c1-mkwqn" Oct 30 22:08:38.981488 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:38.981449 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f344b29-02c0-40ec-96b3-3543ebda4d28-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5a19c1-mkwqn\" (UID: \"3f344b29-02c0-40ec-96b3-3543ebda4d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5a19c1-mkwqn" Oct 30 22:08:38.981543 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:38.981527 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77twb\" (UniqueName: \"kubernetes.io/projected/3f344b29-02c0-40ec-96b3-3543ebda4d28-kube-api-access-77twb\") pod \"maas-api-revoke-keys-e2e-api-b-5a19c1-mkwqn\" (UID: \"3f344b29-02c0-40ec-96b3-3543ebda4d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5a19c1-mkwqn" Oct 30 22:08:39.083124 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:39.083082 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f344b29-02c0-40ec-96b3-3543ebda4d28-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5a19c1-mkwqn\" (UID: \"3f344b29-02c0-40ec-96b3-3543ebda4d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5a19c1-mkwqn" Oct 30 22:08:39.083315 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:39.083157 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f344b29-02c0-40ec-96b3-3543ebda4d28-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5a19c1-mkwqn\" (UID: \"3f344b29-02c0-40ec-96b3-3543ebda4d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5a19c1-mkwqn" Oct 30 22:08:39.083315 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:39.083233 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77twb\" (UniqueName: \"kubernetes.io/projected/3f344b29-02c0-40ec-96b3-3543ebda4d28-kube-api-access-77twb\") pod \"maas-api-revoke-keys-e2e-api-b-5a19c1-mkwqn\" (UID: \"3f344b29-02c0-40ec-96b3-3543ebda4d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5a19c1-mkwqn" Oct 30 22:08:39.083870 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:39.083848 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f344b29-02c0-40ec-96b3-3543ebda4d28-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5a19c1-mkwqn\" (UID: \"3f344b29-02c0-40ec-96b3-3543ebda4d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5a19c1-mkwqn" Oct 30 22:08:39.085536 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:39.085512 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f344b29-02c0-40ec-96b3-3543ebda4d28-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5a19c1-mkwqn\" (UID: \"3f344b29-02c0-40ec-96b3-3543ebda4d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5a19c1-mkwqn" Oct 30 22:08:39.145799 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:39.145758 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77twb\" (UniqueName: \"kubernetes.io/projected/3f344b29-02c0-40ec-96b3-3543ebda4d28-kube-api-access-77twb\") pod \"maas-api-revoke-keys-e2e-api-b-5a19c1-mkwqn\" (UID: \"3f344b29-02c0-40ec-96b3-3543ebda4d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5a19c1-mkwqn" Oct 30 22:08:39.174110 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:39.174078 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5a19c1-mkwqn" Oct 30 22:08:39.343673 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:39.343636 5748 generic.go:356] "Generic (PLEG): container finished" podID="2ee699aa-aa04-4b9f-8f92-4d1c33a15034" containerID="808a60df966cb339b36ba4e5c5c2769023cee8f02c58993d39bdda5c69dfecd4" exitCode=0 Oct 30 22:08:39.343824 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:39.343694 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5bf4f8-ptnj9" event={"ID":"2ee699aa-aa04-4b9f-8f92-4d1c33a15034","Type":"ContainerDied","Data":"808a60df966cb339b36ba4e5c5c2769023cee8f02c58993d39bdda5c69dfecd4"} Oct 30 22:08:39.577935 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:39.577911 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5a19c1-mkwqn"] Oct 30 22:08:39.580016 ip-10-0-143-61 kubenswrapper[5748]: W1030 22:08:39.579987 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f344b29_02c0_40ec_96b3_3543ebda4d28.slice/crio-09d5994d9f143e7605a55e21f8ea9f90ad735183b1fc417cc5334ae205fb87dd WatchSource:0}: Error finding container 09d5994d9f143e7605a55e21f8ea9f90ad735183b1fc417cc5334ae205fb87dd: Status 404 returned error can't find the container with id 09d5994d9f143e7605a55e21f8ea9f90ad735183b1fc417cc5334ae205fb87dd Oct 30 22:08:40.352216 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:40.352119 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5a19c1-mkwqn" event={"ID":"3f344b29-02c0-40ec-96b3-3543ebda4d28","Type":"ContainerStarted","Data":"105101e191f029986dda1afe729cc343ece8ba67342d68c4466a046aa01f59b0"} Oct 30 22:08:40.352216 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:40.352183 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5a19c1-mkwqn" event={"ID":"3f344b29-02c0-40ec-96b3-3543ebda4d28","Type":"ContainerStarted","Data":"09d5994d9f143e7605a55e21f8ea9f90ad735183b1fc417cc5334ae205fb87dd"} Oct 30 22:08:40.492148 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:40.492123 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5bf4f8-ptnj9" Oct 30 22:08:40.601288 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:40.601259 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ee699aa-aa04-4b9f-8f92-4d1c33a15034-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ee699aa-aa04-4b9f-8f92-4d1c33a15034-maas-api-openshift-ca\") pod \"2ee699aa-aa04-4b9f-8f92-4d1c33a15034\" (UID: \"2ee699aa-aa04-4b9f-8f92-4d1c33a15034\") " Oct 30 22:08:40.601439 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:40.601309 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ee699aa-aa04-4b9f-8f92-4d1c33a15034-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ee699aa-aa04-4b9f-8f92-4d1c33a15034-maas-api-xks-ca\") pod \"2ee699aa-aa04-4b9f-8f92-4d1c33a15034\" (UID: \"2ee699aa-aa04-4b9f-8f92-4d1c33a15034\") " Oct 30 22:08:40.601439 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:40.601378 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ee699aa-aa04-4b9f-8f92-4d1c33a15034-kube-api-access-qknm7\" (UniqueName: \"kubernetes.io/projected/2ee699aa-aa04-4b9f-8f92-4d1c33a15034-kube-api-access-qknm7\") pod \"2ee699aa-aa04-4b9f-8f92-4d1c33a15034\" (UID: \"2ee699aa-aa04-4b9f-8f92-4d1c33a15034\") " Oct 30 22:08:40.601648 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:40.601622 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ee699aa-aa04-4b9f-8f92-4d1c33a15034-maas-api-openshift-ca" pod "2ee699aa-aa04-4b9f-8f92-4d1c33a15034" (UID: "2ee699aa-aa04-4b9f-8f92-4d1c33a15034"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:08:40.601711 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:40.601694 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ee699aa-aa04-4b9f-8f92-4d1c33a15034-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:40.603437 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:40.603379 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ee699aa-aa04-4b9f-8f92-4d1c33a15034-maas-api-xks-ca" pod "2ee699aa-aa04-4b9f-8f92-4d1c33a15034" (UID: "2ee699aa-aa04-4b9f-8f92-4d1c33a15034"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:08:40.603531 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:40.603501 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ee699aa-aa04-4b9f-8f92-4d1c33a15034-kube-api-access-qknm7" pod "2ee699aa-aa04-4b9f-8f92-4d1c33a15034" (UID: "2ee699aa-aa04-4b9f-8f92-4d1c33a15034"). InnerVolumeSpecName "kube-api-access-qknm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:40.646588 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:40.646536 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5a19c1-mkwqn" podStartSLOduration=2.6465214870000002 podStartE2EDuration="2.646521487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:40.526913161 +0000 UTC m=+2704.269434087" watchObservedRunningTime="2026-10-30 22:08:40.646521487 +0000 UTC m=+2704.389042463" Oct 30 22:08:40.703340 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:40.703295 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ee699aa-aa04-4b9f-8f92-4d1c33a15034-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:40.703340 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:40.703338 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qknm7\" (UniqueName: \"kubernetes.io/projected/2ee699aa-aa04-4b9f-8f92-4d1c33a15034-kube-api-access-qknm7\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:41.360490 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:41.360453 5748 generic.go:356] "Generic (PLEG): container finished" podID="3f344b29-02c0-40ec-96b3-3543ebda4d28" containerID="105101e191f029986dda1afe729cc343ece8ba67342d68c4466a046aa01f59b0" exitCode=0 Oct 30 22:08:41.360991 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:41.360554 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5a19c1-mkwqn" event={"ID":"3f344b29-02c0-40ec-96b3-3543ebda4d28","Type":"ContainerDied","Data":"105101e191f029986dda1afe729cc343ece8ba67342d68c4466a046aa01f59b0"} Oct 30 22:08:41.361920 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:41.361873 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5bf4f8-ptnj9" Oct 30 22:08:41.362027 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:41.361892 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5bf4f8-ptnj9" event={"ID":"2ee699aa-aa04-4b9f-8f92-4d1c33a15034","Type":"ContainerDied","Data":"6461fbbd6eb7e447115c8bea73377cd0d16826d1396fd4002b4539967ce6ea4a"} Oct 30 22:08:41.362027 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:41.361963 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6461fbbd6eb7e447115c8bea73377cd0d16826d1396fd4002b4539967ce6ea4a" Oct 30 22:08:42.511937 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:42.511911 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5a19c1-mkwqn" Oct 30 22:08:42.624533 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:42.624475 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f344b29-02c0-40ec-96b3-3543ebda4d28-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f344b29-02c0-40ec-96b3-3543ebda4d28-maas-api-xks-ca\") pod \"3f344b29-02c0-40ec-96b3-3543ebda4d28\" (UID: \"3f344b29-02c0-40ec-96b3-3543ebda4d28\") " Oct 30 22:08:42.624654 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:42.624544 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f344b29-02c0-40ec-96b3-3543ebda4d28-kube-api-access-77twb\" (UniqueName: \"kubernetes.io/projected/3f344b29-02c0-40ec-96b3-3543ebda4d28-kube-api-access-77twb\") pod \"3f344b29-02c0-40ec-96b3-3543ebda4d28\" (UID: \"3f344b29-02c0-40ec-96b3-3543ebda4d28\") " Oct 30 22:08:42.624654 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:42.624598 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f344b29-02c0-40ec-96b3-3543ebda4d28-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f344b29-02c0-40ec-96b3-3543ebda4d28-maas-api-openshift-ca\") pod \"3f344b29-02c0-40ec-96b3-3543ebda4d28\" (UID: \"3f344b29-02c0-40ec-96b3-3543ebda4d28\") " Oct 30 22:08:42.625055 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:42.625030 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f344b29-02c0-40ec-96b3-3543ebda4d28-maas-api-openshift-ca" pod "3f344b29-02c0-40ec-96b3-3543ebda4d28" (UID: "3f344b29-02c0-40ec-96b3-3543ebda4d28"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:08:42.626577 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:42.626549 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f344b29-02c0-40ec-96b3-3543ebda4d28-kube-api-access-77twb" pod "3f344b29-02c0-40ec-96b3-3543ebda4d28" (UID: "3f344b29-02c0-40ec-96b3-3543ebda4d28"). InnerVolumeSpecName "kube-api-access-77twb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:42.626651 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:42.626583 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f344b29-02c0-40ec-96b3-3543ebda4d28-maas-api-xks-ca" pod "3f344b29-02c0-40ec-96b3-3543ebda4d28" (UID: "3f344b29-02c0-40ec-96b3-3543ebda4d28"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:08:42.726369 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:42.726332 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f344b29-02c0-40ec-96b3-3543ebda4d28-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:42.726369 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:42.726365 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77twb\" (UniqueName: \"kubernetes.io/projected/3f344b29-02c0-40ec-96b3-3543ebda4d28-kube-api-access-77twb\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:42.726369 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:42.726375 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f344b29-02c0-40ec-96b3-3543ebda4d28-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:43.375290 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:43.375259 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5a19c1-mkwqn" Oct 30 22:08:43.375441 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:43.375266 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5a19c1-mkwqn" event={"ID":"3f344b29-02c0-40ec-96b3-3543ebda4d28","Type":"ContainerDied","Data":"09d5994d9f143e7605a55e21f8ea9f90ad735183b1fc417cc5334ae205fb87dd"} Oct 30 22:08:43.375441 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:43.375419 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09d5994d9f143e7605a55e21f8ea9f90ad735183b1fc417cc5334ae205fb87dd" Oct 30 22:08:51.877953 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:51.877919 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-1a415f-x2js4"] Oct 30 22:08:51.928368 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:51.928336 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-1a415f-x2js4"] Oct 30 22:08:52.416381 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:08:52.416346 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4ce3691-6832-44ca-a3bf-34205dd697b6" path="/var/lib/kubelet/pods/a4ce3691-6832-44ca-a3bf-34205dd697b6/volumes" Oct 30 22:09:00.143844 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:09:00.143808 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:09:20.003374 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:20.003331 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-691598-n2d4g"] Oct 30 22:09:20.003862 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:20.003836 5748 cpu_manager.go:414] "containerMap: removing container" podUID="2ee699aa-aa04-4b9f-8f92-4d1c33a15034" containerName="revoke-keys" Oct 30 22:09:20.003862 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:20.003847 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ee699aa-aa04-4b9f-8f92-4d1c33a15034" containerName="revoke-keys" Oct 30 22:09:20.003962 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:20.003882 5748 cpu_manager.go:414] "containerMap: removing container" podUID="3f344b29-02c0-40ec-96b3-3543ebda4d28" containerName="revoke-keys" Oct 30 22:09:20.003962 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:20.003887 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f344b29-02c0-40ec-96b3-3543ebda4d28" containerName="revoke-keys" Oct 30 22:09:20.004026 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:20.004002 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f344b29-02c0-40ec-96b3-3543ebda4d28" containerName="revoke-keys" Oct 30 22:09:20.004026 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:20.004018 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ee699aa-aa04-4b9f-8f92-4d1c33a15034" containerName="revoke-keys" Oct 30 22:09:20.007686 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:20.007665 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-691598-n2d4g" Oct 30 22:09:20.113300 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:20.113264 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:09:20.113504 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:20.113304 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:09:20.113504 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:20.113381 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 22:09:20.113504 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:20.113392 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:09:20.115921 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:20.115882 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-691598-n2d4g"] Oct 30 22:09:20.130327 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:20.130297 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75d93cb9-cdbe-4756-9350-0a74554c7bf5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-691598-n2d4g\" (UID: \"75d93cb9-cdbe-4756-9350-0a74554c7bf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-691598-n2d4g" Oct 30 22:09:20.130468 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:20.130371 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75d93cb9-cdbe-4756-9350-0a74554c7bf5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-691598-n2d4g\" (UID: \"75d93cb9-cdbe-4756-9350-0a74554c7bf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-691598-n2d4g" Oct 30 22:09:20.130533 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:20.130465 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lzlg\" (UniqueName: \"kubernetes.io/projected/75d93cb9-cdbe-4756-9350-0a74554c7bf5-kube-api-access-2lzlg\") pod \"maas-api-revoke-keys-e2e-worker-w0-691598-n2d4g\" (UID: \"75d93cb9-cdbe-4756-9350-0a74554c7bf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-691598-n2d4g" Oct 30 22:09:20.231634 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:20.231595 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75d93cb9-cdbe-4756-9350-0a74554c7bf5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-691598-n2d4g\" (UID: \"75d93cb9-cdbe-4756-9350-0a74554c7bf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-691598-n2d4g" Oct 30 22:09:20.231634 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:20.231640 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lzlg\" (UniqueName: \"kubernetes.io/projected/75d93cb9-cdbe-4756-9350-0a74554c7bf5-kube-api-access-2lzlg\") pod \"maas-api-revoke-keys-e2e-worker-w0-691598-n2d4g\" (UID: \"75d93cb9-cdbe-4756-9350-0a74554c7bf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-691598-n2d4g" Oct 30 22:09:20.231859 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:20.231726 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75d93cb9-cdbe-4756-9350-0a74554c7bf5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-691598-n2d4g\" (UID: \"75d93cb9-cdbe-4756-9350-0a74554c7bf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-691598-n2d4g" Oct 30 22:09:20.232293 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:20.232263 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75d93cb9-cdbe-4756-9350-0a74554c7bf5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-691598-n2d4g\" (UID: \"75d93cb9-cdbe-4756-9350-0a74554c7bf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-691598-n2d4g" Oct 30 22:09:20.234113 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:20.234089 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75d93cb9-cdbe-4756-9350-0a74554c7bf5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-691598-n2d4g\" (UID: \"75d93cb9-cdbe-4756-9350-0a74554c7bf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-691598-n2d4g" Oct 30 22:09:20.294188 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:20.294119 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lzlg\" (UniqueName: \"kubernetes.io/projected/75d93cb9-cdbe-4756-9350-0a74554c7bf5-kube-api-access-2lzlg\") pod \"maas-api-revoke-keys-e2e-worker-w0-691598-n2d4g\" (UID: \"75d93cb9-cdbe-4756-9350-0a74554c7bf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-691598-n2d4g" Oct 30 22:09:20.321066 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:20.321039 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-691598-n2d4g" Oct 30 22:09:20.697779 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:20.697754 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-691598-n2d4g"] Oct 30 22:09:20.700821 ip-10-0-143-61 kubenswrapper[5748]: W1030 22:09:20.700790 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75d93cb9_cdbe_4756_9350_0a74554c7bf5.slice/crio-dc17cdb1a7876b16566451bc1064ad8d976b389e84c321f22895ae1c1b74410f WatchSource:0}: Error finding container dc17cdb1a7876b16566451bc1064ad8d976b389e84c321f22895ae1c1b74410f: Status 404 returned error can't find the container with id dc17cdb1a7876b16566451bc1064ad8d976b389e84c321f22895ae1c1b74410f Oct 30 22:09:21.603408 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:21.603376 5748 generic.go:356] "Generic (PLEG): container finished" podID="75d93cb9-cdbe-4756-9350-0a74554c7bf5" containerID="f479376610f0c8c94fda5acb5adb19cd6c01e36a17dc0a152766d835127c5e63" exitCode=0 Oct 30 22:09:21.603801 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:21.603457 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-691598-n2d4g" event={"ID":"75d93cb9-cdbe-4756-9350-0a74554c7bf5","Type":"ContainerDied","Data":"f479376610f0c8c94fda5acb5adb19cd6c01e36a17dc0a152766d835127c5e63"} Oct 30 22:09:21.603801 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:21.603507 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-691598-n2d4g" event={"ID":"75d93cb9-cdbe-4756-9350-0a74554c7bf5","Type":"ContainerStarted","Data":"dc17cdb1a7876b16566451bc1064ad8d976b389e84c321f22895ae1c1b74410f"} Oct 30 22:09:22.766102 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:22.766076 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-691598-n2d4g" Oct 30 22:09:22.859818 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:22.859720 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75d93cb9-cdbe-4756-9350-0a74554c7bf5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75d93cb9-cdbe-4756-9350-0a74554c7bf5-maas-api-openshift-ca\") pod \"75d93cb9-cdbe-4756-9350-0a74554c7bf5\" (UID: \"75d93cb9-cdbe-4756-9350-0a74554c7bf5\") " Oct 30 22:09:22.860033 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:22.859823 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75d93cb9-cdbe-4756-9350-0a74554c7bf5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75d93cb9-cdbe-4756-9350-0a74554c7bf5-maas-api-xks-ca\") pod \"75d93cb9-cdbe-4756-9350-0a74554c7bf5\" (UID: \"75d93cb9-cdbe-4756-9350-0a74554c7bf5\") " Oct 30 22:09:22.860033 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:22.859988 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75d93cb9-cdbe-4756-9350-0a74554c7bf5-kube-api-access-2lzlg\" (UniqueName: \"kubernetes.io/projected/75d93cb9-cdbe-4756-9350-0a74554c7bf5-kube-api-access-2lzlg\") pod \"75d93cb9-cdbe-4756-9350-0a74554c7bf5\" (UID: \"75d93cb9-cdbe-4756-9350-0a74554c7bf5\") " Oct 30 22:09:22.860159 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:22.860094 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75d93cb9-cdbe-4756-9350-0a74554c7bf5-maas-api-openshift-ca" pod "75d93cb9-cdbe-4756-9350-0a74554c7bf5" (UID: "75d93cb9-cdbe-4756-9350-0a74554c7bf5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:09:22.860374 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:22.860356 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75d93cb9-cdbe-4756-9350-0a74554c7bf5-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:22.861935 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:22.861887 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75d93cb9-cdbe-4756-9350-0a74554c7bf5-maas-api-xks-ca" pod "75d93cb9-cdbe-4756-9350-0a74554c7bf5" (UID: "75d93cb9-cdbe-4756-9350-0a74554c7bf5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:09:22.862043 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:22.861958 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75d93cb9-cdbe-4756-9350-0a74554c7bf5-kube-api-access-2lzlg" pod "75d93cb9-cdbe-4756-9350-0a74554c7bf5" (UID: "75d93cb9-cdbe-4756-9350-0a74554c7bf5"). InnerVolumeSpecName "kube-api-access-2lzlg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:22.961747 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:22.961712 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75d93cb9-cdbe-4756-9350-0a74554c7bf5-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:22.961747 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:22.961739 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lzlg\" (UniqueName: \"kubernetes.io/projected/75d93cb9-cdbe-4756-9350-0a74554c7bf5-kube-api-access-2lzlg\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:23.616847 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:23.616821 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-691598-n2d4g" Oct 30 22:09:23.617061 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:23.616861 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-691598-n2d4g" event={"ID":"75d93cb9-cdbe-4756-9350-0a74554c7bf5","Type":"ContainerDied","Data":"dc17cdb1a7876b16566451bc1064ad8d976b389e84c321f22895ae1c1b74410f"} Oct 30 22:09:23.617061 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:23.616930 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc17cdb1a7876b16566451bc1064ad8d976b389e84c321f22895ae1c1b74410f" Oct 30 22:09:24.467436 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:24.467403 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-691598/simulator-w0-691598-kserve-76fb5d5bc7-l5gqk"] Oct 30 22:09:30.144670 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:09:30.144636 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:09:31.523565 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:31.523528 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-691598-f774fbb9d-6fxfj"] Oct 30 22:09:31.634102 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:31.634069 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-691598-85b4f65bbc-k4gtg"] Oct 30 22:09:34.038207 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:34.038170 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5bf4f8-ptnj9"] Oct 30 22:09:34.088944 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:34.088910 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5bf4f8-ptnj9"] Oct 30 22:09:34.416238 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:34.416205 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ee699aa-aa04-4b9f-8f92-4d1c33a15034" path="/var/lib/kubelet/pods/2ee699aa-aa04-4b9f-8f92-4d1c33a15034/volumes" Oct 30 22:09:36.524403 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:36.524338 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-691598-f774fbb9d-6fxfj" podUID="3ce6a432-f03c-4e18-abe1-5d5da48d092d" containerName="payload-processing" containerID="cri-o://4f867897fe6047f26cb8c3818dfa4d6f47b0a09ea80ac34b243375dd89b00c4e" gracePeriod=25 Oct 30 22:09:36.635142 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:36.635102 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-691598-85b4f65bbc-k4gtg" podUID="6c34df93-2d70-41c6-9e31-8b7e30fdbe4c" containerName="payload-pre-processing" containerID="cri-o://ec57c92c6963ec711cf9824946ced346e1f27d16ce78f699d24221ddfa39dfb7" gracePeriod=25 Oct 30 22:09:37.521296 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:37.521258 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-691598-85b4f65bbc-k4gtg" podUID="6c34df93-2d70-41c6-9e31-8b7e30fdbe4c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:09:37.521486 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:37.521258 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-691598-f774fbb9d-6fxfj" podUID="3ce6a432-f03c-4e18-abe1-5d5da48d092d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:09:38.267683 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:38.267652 5748 scope.go:122] "RemoveContainer" containerID="bedb2938b537b6b17a2f9aa5850f1342bae65dd4422070dedeb3c4290675bdbb" Oct 30 22:09:38.277230 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:38.277209 5748 scope.go:122] "RemoveContainer" containerID="cc16939db61aef1fe585397b431c3c26de7582d0acd26707c77d41e6acda9007" Oct 30 22:09:39.488377 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:39.488338 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-691598/simulator-w0-691598-kserve-76fb5d5bc7-l5gqk" podUID="4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9" containerName="main" containerID="cri-o://91bb8ff3be5491a8c5744b752374ad9c8f35edae37ca8ea5f864aeecdf424d7f" gracePeriod=45 Oct 30 22:09:39.721325 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:39.721290 5748 generic.go:356] "Generic (PLEG): container finished" podID="4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9" containerID="91bb8ff3be5491a8c5744b752374ad9c8f35edae37ca8ea5f864aeecdf424d7f" exitCode=0 Oct 30 22:09:39.721472 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:39.721353 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-691598/simulator-w0-691598-kserve-76fb5d5bc7-l5gqk" event={"ID":"4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9","Type":"ContainerDied","Data":"91bb8ff3be5491a8c5744b752374ad9c8f35edae37ca8ea5f864aeecdf424d7f"} Oct 30 22:09:39.773958 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:39.773933 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-691598/simulator-w0-691598-kserve-76fb5d5bc7-l5gqk" Oct 30 22:09:39.848809 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:39.848781 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9-tmp-dir\") pod \"4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9\" (UID: \"4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9\") " Oct 30 22:09:39.848985 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:39.848852 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9-kube-api-access-57k8q\" (UniqueName: \"kubernetes.io/projected/4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9-kube-api-access-57k8q\") pod \"4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9\" (UID: \"4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9\") " Oct 30 22:09:39.848985 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:39.848875 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9-model-cache\") pod \"4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9\" (UID: \"4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9\") " Oct 30 22:09:39.848985 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:39.848939 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9-tls-certs\" (UniqueName: \"kubernetes.io/secret/4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9-tls-certs\") pod \"4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9\" (UID: \"4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9\") " Oct 30 22:09:39.849148 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:39.849046 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9-dshm\") pod \"4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9\" (UID: \"4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9\") " Oct 30 22:09:39.849148 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:39.849072 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9-home\" (UniqueName: \"kubernetes.io/empty-dir/4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9-home\") pod \"4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9\" (UID: \"4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9\") " Oct 30 22:09:39.849148 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:39.849072 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9-tmp-dir" pod "4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9" (UID: "4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:39.849323 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:39.849210 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9-model-cache" pod "4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9" (UID: "4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:39.849443 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:39.849414 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9-home" pod "4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9" (UID: "4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:39.849551 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:39.849527 5748 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9-tmp-dir\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:39.849551 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:39.849548 5748 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9-model-cache\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:39.849655 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:39.849563 5748 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9-home\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:39.851280 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:39.851251 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9-dshm" pod "4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9" (UID: "4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:39.851369 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:39.851326 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9-tls-certs" pod "4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9" (UID: "4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:09:39.851415 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:39.851393 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9-kube-api-access-57k8q" pod "4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9" (UID: "4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9"). InnerVolumeSpecName "kube-api-access-57k8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:39.967564 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:39.967527 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57k8q\" (UniqueName: \"kubernetes.io/projected/4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9-kube-api-access-57k8q\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:39.967858 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:39.967838 5748 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9-tls-certs\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:39.967994 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:39.967981 5748 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9-dshm\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:40.728652 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:40.728545 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-691598/simulator-w0-691598-kserve-76fb5d5bc7-l5gqk" Oct 30 22:09:40.729265 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:40.728536 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-691598/simulator-w0-691598-kserve-76fb5d5bc7-l5gqk" event={"ID":"4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9","Type":"ContainerDied","Data":"55821add0564dd4133dad6c0599ef8773b9c2120b3669c74e0cd5995488764c7"} Oct 30 22:09:40.729265 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:40.728709 5748 scope.go:122] "RemoveContainer" containerID="91bb8ff3be5491a8c5744b752374ad9c8f35edae37ca8ea5f864aeecdf424d7f" Oct 30 22:09:40.733543 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:40.733510 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-691598/simulator-w0-691598-kserve-76fb5d5bc7-l5gqk" podUID="4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 30 22:09:40.878847 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:40.878811 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5a19c1-mkwqn"] Oct 30 22:09:40.890004 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:40.889980 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5a19c1-mkwqn"] Oct 30 22:09:40.898525 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:40.898500 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-691598/simulator-w0-691598-kserve-76fb5d5bc7-l5gqk"] Oct 30 22:09:40.929162 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:40.929122 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-691598/simulator-w0-691598-kserve-76fb5d5bc7-l5gqk"] Oct 30 22:09:41.365617 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:41.365576 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj"] Oct 30 22:09:41.365844 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:41.365818 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj" podUID="4bb93eca-1d04-412d-aa83-2e1ac581bf1c" containerName="istio-proxy" containerID="cri-o://4865d5a0e317b2a2978182c6b17a766b0ffab2a5f1471504f1d6aa2e337e9d2c" gracePeriod=30 Oct 30 22:09:42.416260 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:42.416228 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f344b29-02c0-40ec-96b3-3543ebda4d28" path="/var/lib/kubelet/pods/3f344b29-02c0-40ec-96b3-3543ebda4d28/volumes" Oct 30 22:09:42.416652 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:42.416638 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9" path="/var/lib/kubelet/pods/4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9/volumes" Oct 30 22:09:42.506663 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:42.506631 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1de75a-pfz4g"] Oct 30 22:09:42.507137 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:42.507122 5748 cpu_manager.go:414] "containerMap: removing container" podUID="4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9" containerName="main" Oct 30 22:09:42.507186 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:42.507139 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9" containerName="main" Oct 30 22:09:42.507186 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:42.507164 5748 cpu_manager.go:414] "containerMap: removing container" podUID="75d93cb9-cdbe-4756-9350-0a74554c7bf5" containerName="revoke-keys" Oct 30 22:09:42.507186 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:42.507169 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75d93cb9-cdbe-4756-9350-0a74554c7bf5" containerName="revoke-keys" Oct 30 22:09:42.507293 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:42.507282 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cd6c7e1-b6ab-45fc-ac45-e9e3fded21b9" containerName="main" Oct 30 22:09:42.507329 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:42.507300 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="75d93cb9-cdbe-4756-9350-0a74554c7bf5" containerName="revoke-keys" Oct 30 22:09:42.512239 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:42.512219 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1de75a-pfz4g" Oct 30 22:09:42.574658 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:42.574635 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1de75a-pfz4g"] Oct 30 22:09:42.595842 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:42.595817 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b358bd3e-1b9a-4f5a-8c13-6dbb806d4874-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-1de75a-pfz4g\" (UID: \"b358bd3e-1b9a-4f5a-8c13-6dbb806d4874\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1de75a-pfz4g" Oct 30 22:09:42.595987 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:42.595891 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b358bd3e-1b9a-4f5a-8c13-6dbb806d4874-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-1de75a-pfz4g\" (UID: \"b358bd3e-1b9a-4f5a-8c13-6dbb806d4874\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1de75a-pfz4g" Oct 30 22:09:42.596129 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:42.596101 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dx25\" (UniqueName: \"kubernetes.io/projected/b358bd3e-1b9a-4f5a-8c13-6dbb806d4874-kube-api-access-9dx25\") pod \"maas-api-revoke-keys-e2e-worker-w4-1de75a-pfz4g\" (UID: \"b358bd3e-1b9a-4f5a-8c13-6dbb806d4874\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1de75a-pfz4g" Oct 30 22:09:42.621404 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:42.621374 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:09:42.621545 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:42.621479 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 22:09:42.621545 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:42.621488 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:09:42.621545 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:42.621504 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:09:42.697457 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:42.697385 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dx25\" (UniqueName: \"kubernetes.io/projected/b358bd3e-1b9a-4f5a-8c13-6dbb806d4874-kube-api-access-9dx25\") pod \"maas-api-revoke-keys-e2e-worker-w4-1de75a-pfz4g\" (UID: \"b358bd3e-1b9a-4f5a-8c13-6dbb806d4874\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1de75a-pfz4g" Oct 30 22:09:42.697457 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:42.697439 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b358bd3e-1b9a-4f5a-8c13-6dbb806d4874-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-1de75a-pfz4g\" (UID: \"b358bd3e-1b9a-4f5a-8c13-6dbb806d4874\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1de75a-pfz4g" Oct 30 22:09:42.697669 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:42.697554 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b358bd3e-1b9a-4f5a-8c13-6dbb806d4874-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-1de75a-pfz4g\" (UID: \"b358bd3e-1b9a-4f5a-8c13-6dbb806d4874\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1de75a-pfz4g" Oct 30 22:09:42.698275 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:42.698253 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b358bd3e-1b9a-4f5a-8c13-6dbb806d4874-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-1de75a-pfz4g\" (UID: \"b358bd3e-1b9a-4f5a-8c13-6dbb806d4874\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1de75a-pfz4g" Oct 30 22:09:42.699777 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:42.699760 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b358bd3e-1b9a-4f5a-8c13-6dbb806d4874-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-1de75a-pfz4g\" (UID: \"b358bd3e-1b9a-4f5a-8c13-6dbb806d4874\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1de75a-pfz4g" Oct 30 22:09:42.765293 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:42.765268 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dx25\" (UniqueName: \"kubernetes.io/projected/b358bd3e-1b9a-4f5a-8c13-6dbb806d4874-kube-api-access-9dx25\") pod \"maas-api-revoke-keys-e2e-worker-w4-1de75a-pfz4g\" (UID: \"b358bd3e-1b9a-4f5a-8c13-6dbb806d4874\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1de75a-pfz4g" Oct 30 22:09:42.825404 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:42.825377 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1de75a-pfz4g" Oct 30 22:09:43.223092 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:43.223066 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1de75a-pfz4g"] Oct 30 22:09:43.225275 ip-10-0-143-61 kubenswrapper[5748]: W1030 22:09:43.225244 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb358bd3e_1b9a_4f5a_8c13_6dbb806d4874.slice/crio-d1cc54c9d45e072c09ab756d920fa4c53ab745de569d7da641a3db04dae63f7e WatchSource:0}: Error finding container d1cc54c9d45e072c09ab756d920fa4c53ab745de569d7da641a3db04dae63f7e: Status 404 returned error can't find the container with id d1cc54c9d45e072c09ab756d920fa4c53ab745de569d7da641a3db04dae63f7e Oct 30 22:09:43.747431 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:43.747393 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1de75a-pfz4g" event={"ID":"b358bd3e-1b9a-4f5a-8c13-6dbb806d4874","Type":"ContainerStarted","Data":"e7914ef9e59bed0d1770f9497c787a59e602c3fbe1eb71a98f517a76d7ef138f"} Oct 30 22:09:43.747815 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:43.747458 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1de75a-pfz4g" event={"ID":"b358bd3e-1b9a-4f5a-8c13-6dbb806d4874","Type":"ContainerStarted","Data":"d1cc54c9d45e072c09ab756d920fa4c53ab745de569d7da641a3db04dae63f7e"} Oct 30 22:09:43.916015 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:43.915959 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1de75a-pfz4g" podStartSLOduration=1.915943688 podStartE2EDuration="1.915943688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:43.864100325 +0000 UTC m=+2767.606621261" watchObservedRunningTime="2026-10-30 22:09:43.915943688 +0000 UTC m=+2767.658464623" Oct 30 22:09:44.634670 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:44.634639 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d84e8fa-f8zg8"] Oct 30 22:09:44.638483 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:44.638465 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d84e8fa-f8zg8" Oct 30 22:09:44.708514 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:44.708481 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d84e8fa-f8zg8"] Oct 30 22:09:44.724263 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:44.724233 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0ac6c07-ace0-48dd-9922-5497406533e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d84e8fa-f8zg8\" (UID: \"d0ac6c07-ace0-48dd-9922-5497406533e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d84e8fa-f8zg8" Oct 30 22:09:44.724426 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:44.724398 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0ac6c07-ace0-48dd-9922-5497406533e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d84e8fa-f8zg8\" (UID: \"d0ac6c07-ace0-48dd-9922-5497406533e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d84e8fa-f8zg8" Oct 30 22:09:44.724485 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:44.724446 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqbhn\" (UniqueName: \"kubernetes.io/projected/d0ac6c07-ace0-48dd-9922-5497406533e5-kube-api-access-qqbhn\") pod \"maas-api-revoke-sub-8d84e8fa-f8zg8\" (UID: \"d0ac6c07-ace0-48dd-9922-5497406533e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d84e8fa-f8zg8" Oct 30 22:09:44.755782 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:44.755752 5748 generic.go:356] "Generic (PLEG): container finished" podID="b358bd3e-1b9a-4f5a-8c13-6dbb806d4874" containerID="e7914ef9e59bed0d1770f9497c787a59e602c3fbe1eb71a98f517a76d7ef138f" exitCode=0 Oct 30 22:09:44.756183 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:44.755880 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1de75a-pfz4g" event={"ID":"b358bd3e-1b9a-4f5a-8c13-6dbb806d4874","Type":"ContainerDied","Data":"e7914ef9e59bed0d1770f9497c787a59e602c3fbe1eb71a98f517a76d7ef138f"} Oct 30 22:09:44.825333 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:44.825306 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0ac6c07-ace0-48dd-9922-5497406533e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d84e8fa-f8zg8\" (UID: \"d0ac6c07-ace0-48dd-9922-5497406533e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d84e8fa-f8zg8" Oct 30 22:09:44.825458 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:44.825337 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqbhn\" (UniqueName: \"kubernetes.io/projected/d0ac6c07-ace0-48dd-9922-5497406533e5-kube-api-access-qqbhn\") pod \"maas-api-revoke-sub-8d84e8fa-f8zg8\" (UID: \"d0ac6c07-ace0-48dd-9922-5497406533e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d84e8fa-f8zg8" Oct 30 22:09:44.825458 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:44.825410 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0ac6c07-ace0-48dd-9922-5497406533e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d84e8fa-f8zg8\" (UID: \"d0ac6c07-ace0-48dd-9922-5497406533e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d84e8fa-f8zg8" Oct 30 22:09:44.826084 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:44.826059 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0ac6c07-ace0-48dd-9922-5497406533e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d84e8fa-f8zg8\" (UID: \"d0ac6c07-ace0-48dd-9922-5497406533e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d84e8fa-f8zg8" Oct 30 22:09:44.827634 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:44.827612 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0ac6c07-ace0-48dd-9922-5497406533e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d84e8fa-f8zg8\" (UID: \"d0ac6c07-ace0-48dd-9922-5497406533e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d84e8fa-f8zg8" Oct 30 22:09:44.886867 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:44.886816 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqbhn\" (UniqueName: \"kubernetes.io/projected/d0ac6c07-ace0-48dd-9922-5497406533e5-kube-api-access-qqbhn\") pod \"maas-api-revoke-sub-8d84e8fa-f8zg8\" (UID: \"d0ac6c07-ace0-48dd-9922-5497406533e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d84e8fa-f8zg8" Oct 30 22:09:44.951982 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:44.951951 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d84e8fa-f8zg8" Oct 30 22:09:45.330725 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:45.330683 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d84e8fa-f8zg8"] Oct 30 22:09:45.331977 ip-10-0-143-61 kubenswrapper[5748]: W1030 22:09:45.331951 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0ac6c07_ace0_48dd_9922_5497406533e5.slice/crio-7b8481dfbf7550318f4e37c81f94c5195891090753af511d8a04fe6eafb6f632 WatchSource:0}: Error finding container 7b8481dfbf7550318f4e37c81f94c5195891090753af511d8a04fe6eafb6f632: Status 404 returned error can't find the container with id 7b8481dfbf7550318f4e37c81f94c5195891090753af511d8a04fe6eafb6f632 Oct 30 22:09:45.763551 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:45.763516 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d84e8fa-f8zg8" event={"ID":"d0ac6c07-ace0-48dd-9922-5497406533e5","Type":"ContainerStarted","Data":"964d2125e4b150d0b8848f491c1b0150900db97260e035495c920bf353ca089a"} Oct 30 22:09:45.764000 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:45.763574 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d84e8fa-f8zg8" event={"ID":"d0ac6c07-ace0-48dd-9922-5497406533e5","Type":"ContainerStarted","Data":"7b8481dfbf7550318f4e37c81f94c5195891090753af511d8a04fe6eafb6f632"} Oct 30 22:09:45.884655 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:45.884548 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d84e8fa-f8zg8" podStartSLOduration=1.8845291199999998 podStartE2EDuration="1.88452912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:45.882727915 +0000 UTC m=+2769.625248851" watchObservedRunningTime="2026-10-30 22:09:45.88452912 +0000 UTC m=+2769.627050057" Oct 30 22:09:45.917291 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:45.917266 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1de75a-pfz4g" Oct 30 22:09:46.039391 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:46.039354 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b358bd3e-1b9a-4f5a-8c13-6dbb806d4874-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b358bd3e-1b9a-4f5a-8c13-6dbb806d4874-maas-api-xks-ca\") pod \"b358bd3e-1b9a-4f5a-8c13-6dbb806d4874\" (UID: \"b358bd3e-1b9a-4f5a-8c13-6dbb806d4874\") " Oct 30 22:09:46.039391 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:46.039393 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b358bd3e-1b9a-4f5a-8c13-6dbb806d4874-kube-api-access-9dx25\" (UniqueName: \"kubernetes.io/projected/b358bd3e-1b9a-4f5a-8c13-6dbb806d4874-kube-api-access-9dx25\") pod \"b358bd3e-1b9a-4f5a-8c13-6dbb806d4874\" (UID: \"b358bd3e-1b9a-4f5a-8c13-6dbb806d4874\") " Oct 30 22:09:46.039635 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:46.039521 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b358bd3e-1b9a-4f5a-8c13-6dbb806d4874-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b358bd3e-1b9a-4f5a-8c13-6dbb806d4874-maas-api-openshift-ca\") pod \"b358bd3e-1b9a-4f5a-8c13-6dbb806d4874\" (UID: \"b358bd3e-1b9a-4f5a-8c13-6dbb806d4874\") " Oct 30 22:09:46.040576 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:46.040543 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b358bd3e-1b9a-4f5a-8c13-6dbb806d4874-maas-api-openshift-ca" pod "b358bd3e-1b9a-4f5a-8c13-6dbb806d4874" (UID: "b358bd3e-1b9a-4f5a-8c13-6dbb806d4874"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:09:46.042664 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:46.042640 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b358bd3e-1b9a-4f5a-8c13-6dbb806d4874-maas-api-xks-ca" pod "b358bd3e-1b9a-4f5a-8c13-6dbb806d4874" (UID: "b358bd3e-1b9a-4f5a-8c13-6dbb806d4874"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:09:46.042776 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:46.042668 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b358bd3e-1b9a-4f5a-8c13-6dbb806d4874-kube-api-access-9dx25" pod "b358bd3e-1b9a-4f5a-8c13-6dbb806d4874" (UID: "b358bd3e-1b9a-4f5a-8c13-6dbb806d4874"). InnerVolumeSpecName "kube-api-access-9dx25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:46.141309 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:46.141282 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b358bd3e-1b9a-4f5a-8c13-6dbb806d4874-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:46.141309 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:46.141309 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dx25\" (UniqueName: \"kubernetes.io/projected/b358bd3e-1b9a-4f5a-8c13-6dbb806d4874-kube-api-access-9dx25\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:46.141465 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:46.141320 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b358bd3e-1b9a-4f5a-8c13-6dbb806d4874-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:46.610174 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:46.610147 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj" Oct 30 22:09:46.747417 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:46.747345 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-kube-api-access-rz9ct\" (UniqueName: \"kubernetes.io/projected/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-kube-api-access-rz9ct\") pod \"4bb93eca-1d04-412d-aa83-2e1ac581bf1c\" (UID: \"4bb93eca-1d04-412d-aa83-2e1ac581bf1c\") " Oct 30 22:09:46.747578 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:46.747428 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-istio-token\" (UniqueName: \"kubernetes.io/projected/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-istio-token\") pod \"4bb93eca-1d04-412d-aa83-2e1ac581bf1c\" (UID: \"4bb93eca-1d04-412d-aa83-2e1ac581bf1c\") " Oct 30 22:09:46.747578 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:46.747466 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-workload-socket\") pod \"4bb93eca-1d04-412d-aa83-2e1ac581bf1c\" (UID: \"4bb93eca-1d04-412d-aa83-2e1ac581bf1c\") " Oct 30 22:09:46.747578 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:46.747532 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-istio-data\") pod \"4bb93eca-1d04-412d-aa83-2e1ac581bf1c\" (UID: \"4bb93eca-1d04-412d-aa83-2e1ac581bf1c\") " Oct 30 22:09:46.747760 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:46.747585 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-istiod-ca-cert\") pod \"4bb93eca-1d04-412d-aa83-2e1ac581bf1c\" (UID: \"4bb93eca-1d04-412d-aa83-2e1ac581bf1c\") " Oct 30 22:09:46.747760 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:46.747634 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-workload-certs\") pod \"4bb93eca-1d04-412d-aa83-2e1ac581bf1c\" (UID: \"4bb93eca-1d04-412d-aa83-2e1ac581bf1c\") " Oct 30 22:09:46.747760 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:46.747684 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-istio-envoy\") pod \"4bb93eca-1d04-412d-aa83-2e1ac581bf1c\" (UID: \"4bb93eca-1d04-412d-aa83-2e1ac581bf1c\") " Oct 30 22:09:46.747760 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:46.747737 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-istio-podinfo\") pod \"4bb93eca-1d04-412d-aa83-2e1ac581bf1c\" (UID: \"4bb93eca-1d04-412d-aa83-2e1ac581bf1c\") " Oct 30 22:09:46.748038 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:46.747790 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-credential-socket\") pod \"4bb93eca-1d04-412d-aa83-2e1ac581bf1c\" (UID: \"4bb93eca-1d04-412d-aa83-2e1ac581bf1c\") " Oct 30 22:09:46.748038 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:46.747767 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-workload-socket" pod "4bb93eca-1d04-412d-aa83-2e1ac581bf1c" (UID: "4bb93eca-1d04-412d-aa83-2e1ac581bf1c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:46.748038 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:46.747824 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-istio-data" pod "4bb93eca-1d04-412d-aa83-2e1ac581bf1c" (UID: "4bb93eca-1d04-412d-aa83-2e1ac581bf1c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:46.748038 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:46.747947 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-workload-certs" pod "4bb93eca-1d04-412d-aa83-2e1ac581bf1c" (UID: "4bb93eca-1d04-412d-aa83-2e1ac581bf1c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:46.748255 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:46.748039 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-istiod-ca-cert" pod "4bb93eca-1d04-412d-aa83-2e1ac581bf1c" (UID: "4bb93eca-1d04-412d-aa83-2e1ac581bf1c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:09:46.748255 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:46.748223 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-credential-socket" pod "4bb93eca-1d04-412d-aa83-2e1ac581bf1c" (UID: "4bb93eca-1d04-412d-aa83-2e1ac581bf1c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:46.748363 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:46.748351 5748 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-credential-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:46.748421 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:46.748372 5748 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-workload-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:46.748421 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:46.748390 5748 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-istio-data\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:46.748421 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:46.748405 5748 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-istiod-ca-cert\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:46.748527 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:46.748419 5748 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-workload-certs\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:46.749719 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:46.749689 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-istio-token" pod "4bb93eca-1d04-412d-aa83-2e1ac581bf1c" (UID: "4bb93eca-1d04-412d-aa83-2e1ac581bf1c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:46.750033 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:46.750004 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-istio-podinfo" pod "4bb93eca-1d04-412d-aa83-2e1ac581bf1c" (UID: "4bb93eca-1d04-412d-aa83-2e1ac581bf1c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:09:46.750129 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:46.750027 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-istio-envoy" pod "4bb93eca-1d04-412d-aa83-2e1ac581bf1c" (UID: "4bb93eca-1d04-412d-aa83-2e1ac581bf1c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:46.750129 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:46.750051 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-kube-api-access-rz9ct" pod "4bb93eca-1d04-412d-aa83-2e1ac581bf1c" (UID: "4bb93eca-1d04-412d-aa83-2e1ac581bf1c"). InnerVolumeSpecName "kube-api-access-rz9ct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:46.770610 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:46.770585 5748 generic.go:356] "Generic (PLEG): container finished" podID="d0ac6c07-ace0-48dd-9922-5497406533e5" containerID="964d2125e4b150d0b8848f491c1b0150900db97260e035495c920bf353ca089a" exitCode=0 Oct 30 22:09:46.770936 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:46.770669 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d84e8fa-f8zg8" event={"ID":"d0ac6c07-ace0-48dd-9922-5497406533e5","Type":"ContainerDied","Data":"964d2125e4b150d0b8848f491c1b0150900db97260e035495c920bf353ca089a"} Oct 30 22:09:46.771948 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:46.771925 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1de75a-pfz4g" event={"ID":"b358bd3e-1b9a-4f5a-8c13-6dbb806d4874","Type":"ContainerDied","Data":"d1cc54c9d45e072c09ab756d920fa4c53ab745de569d7da641a3db04dae63f7e"} Oct 30 22:09:46.772018 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:46.771945 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1de75a-pfz4g" Oct 30 22:09:46.772018 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:46.771971 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1cc54c9d45e072c09ab756d920fa4c53ab745de569d7da641a3db04dae63f7e" Oct 30 22:09:46.773825 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:46.773800 5748 generic.go:356] "Generic (PLEG): container finished" podID="4bb93eca-1d04-412d-aa83-2e1ac581bf1c" containerID="4865d5a0e317b2a2978182c6b17a766b0ffab2a5f1471504f1d6aa2e337e9d2c" exitCode=0 Oct 30 22:09:46.773970 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:46.773826 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj" event={"ID":"4bb93eca-1d04-412d-aa83-2e1ac581bf1c","Type":"ContainerDied","Data":"4865d5a0e317b2a2978182c6b17a766b0ffab2a5f1471504f1d6aa2e337e9d2c"} Oct 30 22:09:46.773970 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:46.773858 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj" Oct 30 22:09:46.773970 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:46.773866 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj" event={"ID":"4bb93eca-1d04-412d-aa83-2e1ac581bf1c","Type":"ContainerDied","Data":"3aab82d1fa99757cecf549d13cfbc42c0d561ad92ccd132753a82e2757d216e4"} Oct 30 22:09:46.773970 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:46.773917 5748 scope.go:122] "RemoveContainer" containerID="4865d5a0e317b2a2978182c6b17a766b0ffab2a5f1471504f1d6aa2e337e9d2c" Oct 30 22:09:46.787711 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:46.787684 5748 scope.go:122] "RemoveContainer" containerID="bb89cacb82ca662a89e11bea6558fa3b1b31b3ce407467306e5459cf2b88faea" Oct 30 22:09:46.799530 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:46.799513 5748 scope.go:122] "RemoveContainer" containerID="4865d5a0e317b2a2978182c6b17a766b0ffab2a5f1471504f1d6aa2e337e9d2c" Oct 30 22:09:46.799828 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:09:46.799805 5748 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4865d5a0e317b2a2978182c6b17a766b0ffab2a5f1471504f1d6aa2e337e9d2c\": container with ID starting with 4865d5a0e317b2a2978182c6b17a766b0ffab2a5f1471504f1d6aa2e337e9d2c not found: ID does not exist" containerID="4865d5a0e317b2a2978182c6b17a766b0ffab2a5f1471504f1d6aa2e337e9d2c" Oct 30 22:09:46.799945 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:46.799835 5748 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4865d5a0e317b2a2978182c6b17a766b0ffab2a5f1471504f1d6aa2e337e9d2c"} err="failed to get container status \"4865d5a0e317b2a2978182c6b17a766b0ffab2a5f1471504f1d6aa2e337e9d2c\": rpc error: code = NotFound desc = could not find container \"4865d5a0e317b2a2978182c6b17a766b0ffab2a5f1471504f1d6aa2e337e9d2c\": container with ID starting with 4865d5a0e317b2a2978182c6b17a766b0ffab2a5f1471504f1d6aa2e337e9d2c not found: ID does not exist" Oct 30 22:09:46.799945 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:46.799855 5748 scope.go:122] "RemoveContainer" containerID="bb89cacb82ca662a89e11bea6558fa3b1b31b3ce407467306e5459cf2b88faea" Oct 30 22:09:46.800155 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:09:46.800132 5748 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb89cacb82ca662a89e11bea6558fa3b1b31b3ce407467306e5459cf2b88faea\": container with ID starting with bb89cacb82ca662a89e11bea6558fa3b1b31b3ce407467306e5459cf2b88faea not found: ID does not exist" containerID="bb89cacb82ca662a89e11bea6558fa3b1b31b3ce407467306e5459cf2b88faea" Oct 30 22:09:46.800201 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:46.800162 5748 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb89cacb82ca662a89e11bea6558fa3b1b31b3ce407467306e5459cf2b88faea"} err="failed to get container status \"bb89cacb82ca662a89e11bea6558fa3b1b31b3ce407467306e5459cf2b88faea\": rpc error: code = NotFound desc = could not find container \"bb89cacb82ca662a89e11bea6558fa3b1b31b3ce407467306e5459cf2b88faea\": container with ID starting with bb89cacb82ca662a89e11bea6558fa3b1b31b3ce407467306e5459cf2b88faea not found: ID does not exist" Oct 30 22:09:46.849359 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:46.849336 5748 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-istio-token\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:46.849359 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:46.849358 5748 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-istio-envoy\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:46.849503 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:46.849369 5748 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-istio-podinfo\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:46.849503 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:46.849378 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz9ct\" (UniqueName: \"kubernetes.io/projected/4bb93eca-1d04-412d-aa83-2e1ac581bf1c-kube-api-access-rz9ct\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:46.900464 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:46.900440 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj"] Oct 30 22:09:46.905334 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:46.905312 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-5bf4f8-openshift-default-666fbb6497-f8djj"] Oct 30 22:09:46.950723 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:46.950699 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1de75a/simulator-w4-1de75a-kserve-6887dd4b46-kc5c2"] Oct 30 22:09:47.521213 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:47.521167 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-691598-f774fbb9d-6fxfj" podUID="3ce6a432-f03c-4e18-abe1-5d5da48d092d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:09:47.521213 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:47.521181 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-691598-85b4f65bbc-k4gtg" podUID="6c34df93-2d70-41c6-9e31-8b7e30fdbe4c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:09:47.937960 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:47.937929 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d84e8fa-f8zg8" Oct 30 22:09:48.072699 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:48.072619 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0ac6c07-ace0-48dd-9922-5497406533e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0ac6c07-ace0-48dd-9922-5497406533e5-maas-api-xks-ca\") pod \"d0ac6c07-ace0-48dd-9922-5497406533e5\" (UID: \"d0ac6c07-ace0-48dd-9922-5497406533e5\") " Oct 30 22:09:48.072699 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:48.072683 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0ac6c07-ace0-48dd-9922-5497406533e5-kube-api-access-qqbhn\" (UniqueName: \"kubernetes.io/projected/d0ac6c07-ace0-48dd-9922-5497406533e5-kube-api-access-qqbhn\") pod \"d0ac6c07-ace0-48dd-9922-5497406533e5\" (UID: \"d0ac6c07-ace0-48dd-9922-5497406533e5\") " Oct 30 22:09:48.072979 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:48.072705 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0ac6c07-ace0-48dd-9922-5497406533e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0ac6c07-ace0-48dd-9922-5497406533e5-maas-api-openshift-ca\") pod \"d0ac6c07-ace0-48dd-9922-5497406533e5\" (UID: \"d0ac6c07-ace0-48dd-9922-5497406533e5\") " Oct 30 22:09:48.073174 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:48.073149 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0ac6c07-ace0-48dd-9922-5497406533e5-maas-api-openshift-ca" pod "d0ac6c07-ace0-48dd-9922-5497406533e5" (UID: "d0ac6c07-ace0-48dd-9922-5497406533e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:09:48.074782 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:48.074750 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0ac6c07-ace0-48dd-9922-5497406533e5-maas-api-xks-ca" pod "d0ac6c07-ace0-48dd-9922-5497406533e5" (UID: "d0ac6c07-ace0-48dd-9922-5497406533e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:09:48.074947 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:48.074926 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0ac6c07-ace0-48dd-9922-5497406533e5-kube-api-access-qqbhn" pod "d0ac6c07-ace0-48dd-9922-5497406533e5" (UID: "d0ac6c07-ace0-48dd-9922-5497406533e5"). InnerVolumeSpecName "kube-api-access-qqbhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:48.173811 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:48.173782 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0ac6c07-ace0-48dd-9922-5497406533e5-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:48.173811 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:48.173810 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0ac6c07-ace0-48dd-9922-5497406533e5-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:48.174031 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:48.173822 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqbhn\" (UniqueName: \"kubernetes.io/projected/d0ac6c07-ace0-48dd-9922-5497406533e5-kube-api-access-qqbhn\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:48.416355 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:48.416326 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bb93eca-1d04-412d-aa83-2e1ac581bf1c" path="/var/lib/kubelet/pods/4bb93eca-1d04-412d-aa83-2e1ac581bf1c/volumes" Oct 30 22:09:48.790750 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:48.790673 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d84e8fa-f8zg8" Oct 30 22:09:48.790750 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:48.790720 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d84e8fa-f8zg8" event={"ID":"d0ac6c07-ace0-48dd-9922-5497406533e5","Type":"ContainerDied","Data":"7b8481dfbf7550318f4e37c81f94c5195891090753af511d8a04fe6eafb6f632"} Oct 30 22:09:48.791014 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:48.790781 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b8481dfbf7550318f4e37c81f94c5195891090753af511d8a04fe6eafb6f632" Oct 30 22:09:56.056303 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:56.056269 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60cb912d-fn7bk"] Oct 30 22:09:56.106931 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:56.106883 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60cb912d-fn7bk"] Oct 30 22:09:56.417135 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:56.417103 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="994860af-7926-4f82-bd6d-616421b46611" path="/var/lib/kubelet/pods/994860af-7926-4f82-bd6d-616421b46611/volumes" Oct 30 22:09:56.668202 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:56.668138 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-fd2a94/test-model-fd2a94-kserve-678bdc5f47-h4hxc"] Oct 30 22:09:56.837915 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:56.836141 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-691598-f774fbb9d-6fxfj" Oct 30 22:09:56.846959 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:56.844987 5748 generic.go:356] "Generic (PLEG): container finished" podID="3ce6a432-f03c-4e18-abe1-5d5da48d092d" containerID="4f867897fe6047f26cb8c3818dfa4d6f47b0a09ea80ac34b243375dd89b00c4e" exitCode=0 Oct 30 22:09:56.846959 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:56.845034 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-691598-f774fbb9d-6fxfj" Oct 30 22:09:56.846959 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:56.845067 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-691598-f774fbb9d-6fxfj" event={"ID":"3ce6a432-f03c-4e18-abe1-5d5da48d092d","Type":"ContainerDied","Data":"4f867897fe6047f26cb8c3818dfa4d6f47b0a09ea80ac34b243375dd89b00c4e"} Oct 30 22:09:56.846959 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:56.845129 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-691598-f774fbb9d-6fxfj" event={"ID":"3ce6a432-f03c-4e18-abe1-5d5da48d092d","Type":"ContainerDied","Data":"388e86abd9f680fa372f5c6368a0208578baab15c09f544758818caf0487bd59"} Oct 30 22:09:56.846959 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:56.845153 5748 scope.go:122] "RemoveContainer" containerID="4f867897fe6047f26cb8c3818dfa4d6f47b0a09ea80ac34b243375dd89b00c4e" Oct 30 22:09:56.847588 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:56.847460 5748 generic.go:356] "Generic (PLEG): container finished" podID="6c34df93-2d70-41c6-9e31-8b7e30fdbe4c" containerID="ec57c92c6963ec711cf9824946ced346e1f27d16ce78f699d24221ddfa39dfb7" exitCode=0 Oct 30 22:09:56.847588 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:56.847538 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-691598-85b4f65bbc-k4gtg" event={"ID":"6c34df93-2d70-41c6-9e31-8b7e30fdbe4c","Type":"ContainerDied","Data":"ec57c92c6963ec711cf9824946ced346e1f27d16ce78f699d24221ddfa39dfb7"} Oct 30 22:09:56.861397 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:56.861376 5748 scope.go:122] "RemoveContainer" containerID="4f867897fe6047f26cb8c3818dfa4d6f47b0a09ea80ac34b243375dd89b00c4e" Oct 30 22:09:56.861746 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:09:56.861700 5748 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f867897fe6047f26cb8c3818dfa4d6f47b0a09ea80ac34b243375dd89b00c4e\": container with ID starting with 4f867897fe6047f26cb8c3818dfa4d6f47b0a09ea80ac34b243375dd89b00c4e not found: ID does not exist" containerID="4f867897fe6047f26cb8c3818dfa4d6f47b0a09ea80ac34b243375dd89b00c4e" Oct 30 22:09:56.861746 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:56.861728 5748 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f867897fe6047f26cb8c3818dfa4d6f47b0a09ea80ac34b243375dd89b00c4e"} err="failed to get container status \"4f867897fe6047f26cb8c3818dfa4d6f47b0a09ea80ac34b243375dd89b00c4e\": rpc error: code = NotFound desc = could not find container \"4f867897fe6047f26cb8c3818dfa4d6f47b0a09ea80ac34b243375dd89b00c4e\": container with ID starting with 4f867897fe6047f26cb8c3818dfa4d6f47b0a09ea80ac34b243375dd89b00c4e not found: ID does not exist" Oct 30 22:09:56.890977 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:56.889224 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ce6a432-f03c-4e18-abe1-5d5da48d092d-kube-api-access-j7cjk\" (UniqueName: \"kubernetes.io/projected/3ce6a432-f03c-4e18-abe1-5d5da48d092d-kube-api-access-j7cjk\") pod \"3ce6a432-f03c-4e18-abe1-5d5da48d092d\" (UID: \"3ce6a432-f03c-4e18-abe1-5d5da48d092d\") " Oct 30 22:09:56.890977 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:56.889295 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ce6a432-f03c-4e18-abe1-5d5da48d092d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ce6a432-f03c-4e18-abe1-5d5da48d092d-plugins-config-volume\") pod \"3ce6a432-f03c-4e18-abe1-5d5da48d092d\" (UID: \"3ce6a432-f03c-4e18-abe1-5d5da48d092d\") " Oct 30 22:09:56.891442 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:56.891399 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ce6a432-f03c-4e18-abe1-5d5da48d092d-plugins-config-volume" pod "3ce6a432-f03c-4e18-abe1-5d5da48d092d" (UID: "3ce6a432-f03c-4e18-abe1-5d5da48d092d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:09:56.893194 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:56.893153 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ce6a432-f03c-4e18-abe1-5d5da48d092d-kube-api-access-j7cjk" pod "3ce6a432-f03c-4e18-abe1-5d5da48d092d" (UID: "3ce6a432-f03c-4e18-abe1-5d5da48d092d"). InnerVolumeSpecName "kube-api-access-j7cjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:56.968631 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:56.968605 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-691598-85b4f65bbc-k4gtg" Oct 30 22:09:56.990087 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:56.990066 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c34df93-2d70-41c6-9e31-8b7e30fdbe4c-kube-api-access-lvrt5\" (UniqueName: \"kubernetes.io/projected/6c34df93-2d70-41c6-9e31-8b7e30fdbe4c-kube-api-access-lvrt5\") pod \"6c34df93-2d70-41c6-9e31-8b7e30fdbe4c\" (UID: \"6c34df93-2d70-41c6-9e31-8b7e30fdbe4c\") " Oct 30 22:09:56.990170 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:56.990139 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c34df93-2d70-41c6-9e31-8b7e30fdbe4c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c34df93-2d70-41c6-9e31-8b7e30fdbe4c-plugins-config-volume\") pod \"6c34df93-2d70-41c6-9e31-8b7e30fdbe4c\" (UID: \"6c34df93-2d70-41c6-9e31-8b7e30fdbe4c\") " Oct 30 22:09:56.990397 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:56.990385 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7cjk\" (UniqueName: \"kubernetes.io/projected/3ce6a432-f03c-4e18-abe1-5d5da48d092d-kube-api-access-j7cjk\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:56.990452 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:56.990399 5748 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ce6a432-f03c-4e18-abe1-5d5da48d092d-plugins-config-volume\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:56.990489 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:56.990469 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c34df93-2d70-41c6-9e31-8b7e30fdbe4c-plugins-config-volume" pod "6c34df93-2d70-41c6-9e31-8b7e30fdbe4c" (UID: "6c34df93-2d70-41c6-9e31-8b7e30fdbe4c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:09:56.992170 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:56.992149 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c34df93-2d70-41c6-9e31-8b7e30fdbe4c-kube-api-access-lvrt5" pod "6c34df93-2d70-41c6-9e31-8b7e30fdbe4c" (UID: "6c34df93-2d70-41c6-9e31-8b7e30fdbe4c"). InnerVolumeSpecName "kube-api-access-lvrt5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:57.091184 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:57.091147 5748 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c34df93-2d70-41c6-9e31-8b7e30fdbe4c-plugins-config-volume\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:57.091184 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:57.091183 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvrt5\" (UniqueName: \"kubernetes.io/projected/6c34df93-2d70-41c6-9e31-8b7e30fdbe4c-kube-api-access-lvrt5\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:57.288714 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:57.288629 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-691598-f774fbb9d-6fxfj"] Oct 30 22:09:57.290781 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:57.290754 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-691598-f774fbb9d-6fxfj"] Oct 30 22:09:57.757272 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:57.757233 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5f78b46-w24k6"] Oct 30 22:09:57.757953 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:57.757932 5748 cpu_manager.go:414] "containerMap: removing container" podUID="4bb93eca-1d04-412d-aa83-2e1ac581bf1c" containerName="istio-proxy" Oct 30 22:09:57.757953 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:57.757951 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bb93eca-1d04-412d-aa83-2e1ac581bf1c" containerName="istio-proxy" Oct 30 22:09:57.758193 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:57.757968 5748 cpu_manager.go:414] "containerMap: removing container" podUID="b358bd3e-1b9a-4f5a-8c13-6dbb806d4874" containerName="revoke-keys" Oct 30 22:09:57.758193 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:57.757977 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b358bd3e-1b9a-4f5a-8c13-6dbb806d4874" containerName="revoke-keys" Oct 30 22:09:57.758193 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:57.758017 5748 cpu_manager.go:414] "containerMap: removing container" podUID="3ce6a432-f03c-4e18-abe1-5d5da48d092d" containerName="payload-processing" Oct 30 22:09:57.758193 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:57.758027 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ce6a432-f03c-4e18-abe1-5d5da48d092d" containerName="payload-processing" Oct 30 22:09:57.758193 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:57.758038 5748 cpu_manager.go:414] "containerMap: removing container" podUID="4bb93eca-1d04-412d-aa83-2e1ac581bf1c" containerName="istio-proxy" Oct 30 22:09:57.758193 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:57.758045 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bb93eca-1d04-412d-aa83-2e1ac581bf1c" containerName="istio-proxy" Oct 30 22:09:57.758193 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:57.758057 5748 cpu_manager.go:414] "containerMap: removing container" podUID="d0ac6c07-ace0-48dd-9922-5497406533e5" containerName="revoke-keys" Oct 30 22:09:57.758193 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:57.758064 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0ac6c07-ace0-48dd-9922-5497406533e5" containerName="revoke-keys" Oct 30 22:09:57.758193 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:57.758083 5748 cpu_manager.go:414] "containerMap: removing container" podUID="6c34df93-2d70-41c6-9e31-8b7e30fdbe4c" containerName="payload-pre-processing" Oct 30 22:09:57.758193 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:57.758091 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c34df93-2d70-41c6-9e31-8b7e30fdbe4c" containerName="payload-pre-processing" Oct 30 22:09:57.758193 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:57.758193 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ce6a432-f03c-4e18-abe1-5d5da48d092d" containerName="payload-processing" Oct 30 22:09:57.758568 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:57.758206 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bb93eca-1d04-412d-aa83-2e1ac581bf1c" containerName="istio-proxy" Oct 30 22:09:57.758568 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:57.758215 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0ac6c07-ace0-48dd-9922-5497406533e5" containerName="revoke-keys" Oct 30 22:09:57.758568 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:57.758227 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c34df93-2d70-41c6-9e31-8b7e30fdbe4c" containerName="payload-pre-processing" Oct 30 22:09:57.758568 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:57.758241 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bb93eca-1d04-412d-aa83-2e1ac581bf1c" containerName="istio-proxy" Oct 30 22:09:57.758568 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:57.758249 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="b358bd3e-1b9a-4f5a-8c13-6dbb806d4874" containerName="revoke-keys" Oct 30 22:09:57.763926 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:57.763390 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f78b46-w24k6" Oct 30 22:09:57.819461 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:57.819382 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:09:57.819461 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:57.819400 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5f78b46-w24k6"] Oct 30 22:09:57.823283 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:57.823140 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:09:57.823283 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:57.823169 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 22:09:57.826593 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:57.826442 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:09:57.856002 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:57.855974 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-691598-85b4f65bbc-k4gtg" event={"ID":"6c34df93-2d70-41c6-9e31-8b7e30fdbe4c","Type":"ContainerDied","Data":"ca452dc71fe19c8796cf7258b79c0fbe4bdbeca8416fbf20b1522d3f0e99a770"} Oct 30 22:09:57.856134 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:57.856026 5748 scope.go:122] "RemoveContainer" containerID="ec57c92c6963ec711cf9824946ced346e1f27d16ce78f699d24221ddfa39dfb7" Oct 30 22:09:57.856183 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:57.856137 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-691598-85b4f65bbc-k4gtg" Oct 30 22:09:57.899423 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:57.899394 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd995578-e1ab-4647-8092-8ddf357b2060-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5f78b46-w24k6\" (UID: \"dd995578-e1ab-4647-8092-8ddf357b2060\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f78b46-w24k6" Oct 30 22:09:57.899575 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:57.899440 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd995578-e1ab-4647-8092-8ddf357b2060-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5f78b46-w24k6\" (UID: \"dd995578-e1ab-4647-8092-8ddf357b2060\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f78b46-w24k6" Oct 30 22:09:57.899575 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:57.899549 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95zzp\" (UniqueName: \"kubernetes.io/projected/dd995578-e1ab-4647-8092-8ddf357b2060-kube-api-access-95zzp\") pod \"maas-api-revoke-sub-a5f78b46-w24k6\" (UID: \"dd995578-e1ab-4647-8092-8ddf357b2060\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f78b46-w24k6" Oct 30 22:09:58.000695 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:58.000660 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd995578-e1ab-4647-8092-8ddf357b2060-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5f78b46-w24k6\" (UID: \"dd995578-e1ab-4647-8092-8ddf357b2060\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f78b46-w24k6" Oct 30 22:09:58.000870 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:58.000701 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd995578-e1ab-4647-8092-8ddf357b2060-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5f78b46-w24k6\" (UID: \"dd995578-e1ab-4647-8092-8ddf357b2060\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f78b46-w24k6" Oct 30 22:09:58.000870 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:58.000742 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95zzp\" (UniqueName: \"kubernetes.io/projected/dd995578-e1ab-4647-8092-8ddf357b2060-kube-api-access-95zzp\") pod \"maas-api-revoke-sub-a5f78b46-w24k6\" (UID: \"dd995578-e1ab-4647-8092-8ddf357b2060\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f78b46-w24k6" Oct 30 22:09:58.001457 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:58.001430 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd995578-e1ab-4647-8092-8ddf357b2060-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5f78b46-w24k6\" (UID: \"dd995578-e1ab-4647-8092-8ddf357b2060\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f78b46-w24k6" Oct 30 22:09:58.002983 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:58.002954 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd995578-e1ab-4647-8092-8ddf357b2060-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5f78b46-w24k6\" (UID: \"dd995578-e1ab-4647-8092-8ddf357b2060\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f78b46-w24k6" Oct 30 22:09:58.019940 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:58.019909 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-691598-85b4f65bbc-k4gtg"] Oct 30 22:09:58.021906 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:58.021876 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-691598-85b4f65bbc-k4gtg"] Oct 30 22:09:58.062688 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:58.062665 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95zzp\" (UniqueName: \"kubernetes.io/projected/dd995578-e1ab-4647-8092-8ddf357b2060-kube-api-access-95zzp\") pod \"maas-api-revoke-sub-a5f78b46-w24k6\" (UID: \"dd995578-e1ab-4647-8092-8ddf357b2060\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f78b46-w24k6" Oct 30 22:09:58.078570 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:58.078513 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f78b46-w24k6" Oct 30 22:09:58.417231 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:58.417199 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ce6a432-f03c-4e18-abe1-5d5da48d092d" path="/var/lib/kubelet/pods/3ce6a432-f03c-4e18-abe1-5d5da48d092d/volumes" Oct 30 22:09:58.417614 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:58.417564 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c34df93-2d70-41c6-9e31-8b7e30fdbe4c" path="/var/lib/kubelet/pods/6c34df93-2d70-41c6-9e31-8b7e30fdbe4c/volumes" Oct 30 22:09:58.457445 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:58.457422 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5f78b46-w24k6"] Oct 30 22:09:58.459390 ip-10-0-143-61 kubenswrapper[5748]: W1030 22:09:58.459365 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd995578_e1ab_4647_8092_8ddf357b2060.slice/crio-2aaae721cecfe413d1e74353a8f20f136819e3818aaf07d28576a13fe40db72d WatchSource:0}: Error finding container 2aaae721cecfe413d1e74353a8f20f136819e3818aaf07d28576a13fe40db72d: Status 404 returned error can't find the container with id 2aaae721cecfe413d1e74353a8f20f136819e3818aaf07d28576a13fe40db72d Oct 30 22:09:58.862724 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:58.862680 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f78b46-w24k6" event={"ID":"dd995578-e1ab-4647-8092-8ddf357b2060","Type":"ContainerStarted","Data":"f50b657fa217395911ec06400f51900f5168eff0f1aa54ad1d32969c12b09b12"} Oct 30 22:09:58.862952 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:58.862740 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f78b46-w24k6" event={"ID":"dd995578-e1ab-4647-8092-8ddf357b2060","Type":"ContainerStarted","Data":"2aaae721cecfe413d1e74353a8f20f136819e3818aaf07d28576a13fe40db72d"} Oct 30 22:09:58.981213 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:58.981121 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f78b46-w24k6" podStartSLOduration=1.9811077400000001 podStartE2EDuration="1.98110774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:58.979313711 +0000 UTC m=+2782.721834648" watchObservedRunningTime="2026-10-30 22:09:58.98110774 +0000 UTC m=+2782.723628669" Oct 30 22:09:59.870883 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:59.870850 5748 generic.go:356] "Generic (PLEG): container finished" podID="dd995578-e1ab-4647-8092-8ddf357b2060" containerID="f50b657fa217395911ec06400f51900f5168eff0f1aa54ad1d32969c12b09b12" exitCode=0 Oct 30 22:09:59.871329 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:09:59.870939 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f78b46-w24k6" event={"ID":"dd995578-e1ab-4647-8092-8ddf357b2060","Type":"ContainerDied","Data":"f50b657fa217395911ec06400f51900f5168eff0f1aa54ad1d32969c12b09b12"} Oct 30 22:10:00.144616 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:10:00.144576 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:10:00.201638 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:00.201603 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6ea374-gbn2r"] Oct 30 22:10:00.205335 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:00.205316 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6ea374-gbn2r" Oct 30 22:10:00.225881 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:00.225851 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/665fa115-464b-4259-9f81-a2907f42db04-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-6ea374-gbn2r\" (UID: \"665fa115-464b-4259-9f81-a2907f42db04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6ea374-gbn2r" Oct 30 22:10:00.226042 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:00.225891 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rccdl\" (UniqueName: \"kubernetes.io/projected/665fa115-464b-4259-9f81-a2907f42db04-kube-api-access-rccdl\") pod \"maas-api-revoke-keys-e2e-worker-w1-6ea374-gbn2r\" (UID: \"665fa115-464b-4259-9f81-a2907f42db04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6ea374-gbn2r" Oct 30 22:10:00.226086 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:00.226044 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/665fa115-464b-4259-9f81-a2907f42db04-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-6ea374-gbn2r\" (UID: \"665fa115-464b-4259-9f81-a2907f42db04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6ea374-gbn2r" Oct 30 22:10:00.266567 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:00.266533 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6ea374-gbn2r"] Oct 30 22:10:00.327132 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:00.327091 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/665fa115-464b-4259-9f81-a2907f42db04-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-6ea374-gbn2r\" (UID: \"665fa115-464b-4259-9f81-a2907f42db04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6ea374-gbn2r" Oct 30 22:10:00.327311 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:00.327192 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/665fa115-464b-4259-9f81-a2907f42db04-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-6ea374-gbn2r\" (UID: \"665fa115-464b-4259-9f81-a2907f42db04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6ea374-gbn2r" Oct 30 22:10:00.327311 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:00.327221 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rccdl\" (UniqueName: \"kubernetes.io/projected/665fa115-464b-4259-9f81-a2907f42db04-kube-api-access-rccdl\") pod \"maas-api-revoke-keys-e2e-worker-w1-6ea374-gbn2r\" (UID: \"665fa115-464b-4259-9f81-a2907f42db04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6ea374-gbn2r" Oct 30 22:10:00.327855 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:00.327832 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/665fa115-464b-4259-9f81-a2907f42db04-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-6ea374-gbn2r\" (UID: \"665fa115-464b-4259-9f81-a2907f42db04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6ea374-gbn2r" Oct 30 22:10:00.329481 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:00.329464 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/665fa115-464b-4259-9f81-a2907f42db04-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-6ea374-gbn2r\" (UID: \"665fa115-464b-4259-9f81-a2907f42db04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6ea374-gbn2r" Oct 30 22:10:00.389157 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:00.389119 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rccdl\" (UniqueName: \"kubernetes.io/projected/665fa115-464b-4259-9f81-a2907f42db04-kube-api-access-rccdl\") pod \"maas-api-revoke-keys-e2e-worker-w1-6ea374-gbn2r\" (UID: \"665fa115-464b-4259-9f81-a2907f42db04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6ea374-gbn2r" Oct 30 22:10:00.518982 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:00.518882 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6ea374-gbn2r" Oct 30 22:10:00.902705 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:00.902677 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6ea374-gbn2r"] Oct 30 22:10:00.903147 ip-10-0-143-61 kubenswrapper[5748]: W1030 22:10:00.902971 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod665fa115_464b_4259_9f81_a2907f42db04.slice/crio-479589d390e1a17232e01cff8905076129203fb0074a7afd1f7d74d80e205435 WatchSource:0}: Error finding container 479589d390e1a17232e01cff8905076129203fb0074a7afd1f7d74d80e205435: Status 404 returned error can't find the container with id 479589d390e1a17232e01cff8905076129203fb0074a7afd1f7d74d80e205435 Oct 30 22:10:00.998866 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:00.998838 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f78b46-w24k6" Oct 30 22:10:01.034296 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:01.034256 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd995578-e1ab-4647-8092-8ddf357b2060-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd995578-e1ab-4647-8092-8ddf357b2060-maas-api-openshift-ca\") pod \"dd995578-e1ab-4647-8092-8ddf357b2060\" (UID: \"dd995578-e1ab-4647-8092-8ddf357b2060\") " Oct 30 22:10:01.034508 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:01.034490 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd995578-e1ab-4647-8092-8ddf357b2060-kube-api-access-95zzp\" (UniqueName: \"kubernetes.io/projected/dd995578-e1ab-4647-8092-8ddf357b2060-kube-api-access-95zzp\") pod \"dd995578-e1ab-4647-8092-8ddf357b2060\" (UID: \"dd995578-e1ab-4647-8092-8ddf357b2060\") " Oct 30 22:10:01.034574 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:01.034560 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd995578-e1ab-4647-8092-8ddf357b2060-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd995578-e1ab-4647-8092-8ddf357b2060-maas-api-xks-ca\") pod \"dd995578-e1ab-4647-8092-8ddf357b2060\" (UID: \"dd995578-e1ab-4647-8092-8ddf357b2060\") " Oct 30 22:10:01.035163 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:01.035086 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd995578-e1ab-4647-8092-8ddf357b2060-maas-api-openshift-ca" pod "dd995578-e1ab-4647-8092-8ddf357b2060" (UID: "dd995578-e1ab-4647-8092-8ddf357b2060"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:10:01.037443 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:01.037403 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd995578-e1ab-4647-8092-8ddf357b2060-maas-api-xks-ca" pod "dd995578-e1ab-4647-8092-8ddf357b2060" (UID: "dd995578-e1ab-4647-8092-8ddf357b2060"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:10:01.039869 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:01.039839 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd995578-e1ab-4647-8092-8ddf357b2060-kube-api-access-95zzp" pod "dd995578-e1ab-4647-8092-8ddf357b2060" (UID: "dd995578-e1ab-4647-8092-8ddf357b2060"). InnerVolumeSpecName "kube-api-access-95zzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:01.136438 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:01.136349 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95zzp\" (UniqueName: \"kubernetes.io/projected/dd995578-e1ab-4647-8092-8ddf357b2060-kube-api-access-95zzp\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:01.136438 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:01.136379 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd995578-e1ab-4647-8092-8ddf357b2060-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:01.136438 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:01.136390 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd995578-e1ab-4647-8092-8ddf357b2060-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:01.885607 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:01.885570 5748 generic.go:356] "Generic (PLEG): container finished" podID="665fa115-464b-4259-9f81-a2907f42db04" containerID="7db59b582526e2ec6fa0ca9353423290bdc117d776073e06523dadb4ce3e6d1f" exitCode=0 Oct 30 22:10:01.885789 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:01.885684 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6ea374-gbn2r" event={"ID":"665fa115-464b-4259-9f81-a2907f42db04","Type":"ContainerDied","Data":"7db59b582526e2ec6fa0ca9353423290bdc117d776073e06523dadb4ce3e6d1f"} Oct 30 22:10:01.885789 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:01.885733 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6ea374-gbn2r" event={"ID":"665fa115-464b-4259-9f81-a2907f42db04","Type":"ContainerStarted","Data":"479589d390e1a17232e01cff8905076129203fb0074a7afd1f7d74d80e205435"} Oct 30 22:10:01.887002 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:01.886980 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f78b46-w24k6" Oct 30 22:10:01.887121 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:01.887017 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f78b46-w24k6" event={"ID":"dd995578-e1ab-4647-8092-8ddf357b2060","Type":"ContainerDied","Data":"2aaae721cecfe413d1e74353a8f20f136819e3818aaf07d28576a13fe40db72d"} Oct 30 22:10:01.887121 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:01.887055 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2aaae721cecfe413d1e74353a8f20f136819e3818aaf07d28576a13fe40db72d" Oct 30 22:10:01.972067 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:01.971958 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-1de75a/simulator-w4-1de75a-kserve-6887dd4b46-kc5c2" podUID="926fa847-0c3c-43e0-b378-04c82433cd98" containerName="main" containerID="cri-o://efb462192c8e0970e0ba9df89d040031ba835fb0074cfd84ae03e3ce7349c9e6" gracePeriod=45 Oct 30 22:10:01.995631 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:10:01.995602 5748 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod926fa847_0c3c_43e0_b378_04c82433cd98.slice/crio-efb462192c8e0970e0ba9df89d040031ba835fb0074cfd84ae03e3ce7349c9e6.scope\": RecentStats: unable to find data in memory cache]" Oct 30 22:10:02.234291 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:02.234229 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1de75a/simulator-w4-1de75a-kserve-6887dd4b46-kc5c2" Oct 30 22:10:02.249549 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:02.249517 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/926fa847-0c3c-43e0-b378-04c82433cd98-home\" (UniqueName: \"kubernetes.io/empty-dir/926fa847-0c3c-43e0-b378-04c82433cd98-home\") pod \"926fa847-0c3c-43e0-b378-04c82433cd98\" (UID: \"926fa847-0c3c-43e0-b378-04c82433cd98\") " Oct 30 22:10:02.249694 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:02.249577 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/926fa847-0c3c-43e0-b378-04c82433cd98-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/926fa847-0c3c-43e0-b378-04c82433cd98-model-cache\") pod \"926fa847-0c3c-43e0-b378-04c82433cd98\" (UID: \"926fa847-0c3c-43e0-b378-04c82433cd98\") " Oct 30 22:10:02.249772 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:02.249753 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/926fa847-0c3c-43e0-b378-04c82433cd98-tls-certs\" (UniqueName: \"kubernetes.io/secret/926fa847-0c3c-43e0-b378-04c82433cd98-tls-certs\") pod \"926fa847-0c3c-43e0-b378-04c82433cd98\" (UID: \"926fa847-0c3c-43e0-b378-04c82433cd98\") " Oct 30 22:10:02.249829 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:02.249783 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/926fa847-0c3c-43e0-b378-04c82433cd98-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/926fa847-0c3c-43e0-b378-04c82433cd98-tmp-dir\") pod \"926fa847-0c3c-43e0-b378-04c82433cd98\" (UID: \"926fa847-0c3c-43e0-b378-04c82433cd98\") " Oct 30 22:10:02.249829 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:02.249782 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/926fa847-0c3c-43e0-b378-04c82433cd98-home" pod "926fa847-0c3c-43e0-b378-04c82433cd98" (UID: "926fa847-0c3c-43e0-b378-04c82433cd98"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:02.249829 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:02.249823 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/926fa847-0c3c-43e0-b378-04c82433cd98-dshm\" (UniqueName: \"kubernetes.io/empty-dir/926fa847-0c3c-43e0-b378-04c82433cd98-dshm\") pod \"926fa847-0c3c-43e0-b378-04c82433cd98\" (UID: \"926fa847-0c3c-43e0-b378-04c82433cd98\") " Oct 30 22:10:02.250048 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:02.249853 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/926fa847-0c3c-43e0-b378-04c82433cd98-model-cache" pod "926fa847-0c3c-43e0-b378-04c82433cd98" (UID: "926fa847-0c3c-43e0-b378-04c82433cd98"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:02.250048 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:02.249888 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/926fa847-0c3c-43e0-b378-04c82433cd98-kube-api-access-kfqvl\" (UniqueName: \"kubernetes.io/projected/926fa847-0c3c-43e0-b378-04c82433cd98-kube-api-access-kfqvl\") pod \"926fa847-0c3c-43e0-b378-04c82433cd98\" (UID: \"926fa847-0c3c-43e0-b378-04c82433cd98\") " Oct 30 22:10:02.250163 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:02.250066 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/926fa847-0c3c-43e0-b378-04c82433cd98-tmp-dir" pod "926fa847-0c3c-43e0-b378-04c82433cd98" (UID: "926fa847-0c3c-43e0-b378-04c82433cd98"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:02.250353 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:02.250327 5748 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/926fa847-0c3c-43e0-b378-04c82433cd98-tmp-dir\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:02.250353 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:02.250355 5748 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/926fa847-0c3c-43e0-b378-04c82433cd98-home\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:02.250535 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:02.250370 5748 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/926fa847-0c3c-43e0-b378-04c82433cd98-model-cache\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:02.252162 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:02.252140 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/926fa847-0c3c-43e0-b378-04c82433cd98-tls-certs" pod "926fa847-0c3c-43e0-b378-04c82433cd98" (UID: "926fa847-0c3c-43e0-b378-04c82433cd98"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:10:02.252384 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:02.252369 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/926fa847-0c3c-43e0-b378-04c82433cd98-dshm" pod "926fa847-0c3c-43e0-b378-04c82433cd98" (UID: "926fa847-0c3c-43e0-b378-04c82433cd98"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:02.252595 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:02.252572 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/926fa847-0c3c-43e0-b378-04c82433cd98-kube-api-access-kfqvl" pod "926fa847-0c3c-43e0-b378-04c82433cd98" (UID: "926fa847-0c3c-43e0-b378-04c82433cd98"). InnerVolumeSpecName "kube-api-access-kfqvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:02.351456 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:02.351425 5748 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/926fa847-0c3c-43e0-b378-04c82433cd98-tls-certs\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:02.351456 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:02.351451 5748 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/926fa847-0c3c-43e0-b378-04c82433cd98-dshm\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:02.351621 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:02.351462 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfqvl\" (UniqueName: \"kubernetes.io/projected/926fa847-0c3c-43e0-b378-04c82433cd98-kube-api-access-kfqvl\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:02.895704 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:02.895666 5748 generic.go:356] "Generic (PLEG): container finished" podID="926fa847-0c3c-43e0-b378-04c82433cd98" containerID="efb462192c8e0970e0ba9df89d040031ba835fb0074cfd84ae03e3ce7349c9e6" exitCode=0 Oct 30 22:10:02.895885 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:02.895714 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1de75a/simulator-w4-1de75a-kserve-6887dd4b46-kc5c2" Oct 30 22:10:02.895885 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:02.895758 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1de75a/simulator-w4-1de75a-kserve-6887dd4b46-kc5c2" event={"ID":"926fa847-0c3c-43e0-b378-04c82433cd98","Type":"ContainerDied","Data":"efb462192c8e0970e0ba9df89d040031ba835fb0074cfd84ae03e3ce7349c9e6"} Oct 30 22:10:02.895885 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:02.895810 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1de75a/simulator-w4-1de75a-kserve-6887dd4b46-kc5c2" event={"ID":"926fa847-0c3c-43e0-b378-04c82433cd98","Type":"ContainerDied","Data":"b063ab1039352549a6eb78f71e89ccf72a9605c2864c0b045958912746ada33f"} Oct 30 22:10:02.895885 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:02.895834 5748 scope.go:122] "RemoveContainer" containerID="efb462192c8e0970e0ba9df89d040031ba835fb0074cfd84ae03e3ce7349c9e6" Oct 30 22:10:02.906074 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:02.906037 5748 scope.go:122] "RemoveContainer" containerID="efb462192c8e0970e0ba9df89d040031ba835fb0074cfd84ae03e3ce7349c9e6" Oct 30 22:10:02.906390 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:10:02.906365 5748 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efb462192c8e0970e0ba9df89d040031ba835fb0074cfd84ae03e3ce7349c9e6\": container with ID starting with efb462192c8e0970e0ba9df89d040031ba835fb0074cfd84ae03e3ce7349c9e6 not found: ID does not exist" containerID="efb462192c8e0970e0ba9df89d040031ba835fb0074cfd84ae03e3ce7349c9e6" Oct 30 22:10:02.906497 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:02.906399 5748 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efb462192c8e0970e0ba9df89d040031ba835fb0074cfd84ae03e3ce7349c9e6"} err="failed to get container status \"efb462192c8e0970e0ba9df89d040031ba835fb0074cfd84ae03e3ce7349c9e6\": rpc error: code = NotFound desc = could not find container \"efb462192c8e0970e0ba9df89d040031ba835fb0074cfd84ae03e3ce7349c9e6\": container with ID starting with efb462192c8e0970e0ba9df89d040031ba835fb0074cfd84ae03e3ce7349c9e6 not found: ID does not exist" Oct 30 22:10:03.096617 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:03.096444 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1de75a/simulator-w4-1de75a-kserve-6887dd4b46-kc5c2"] Oct 30 22:10:03.101272 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:03.101246 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-1de75a/simulator-w4-1de75a-kserve-6887dd4b46-kc5c2"] Oct 30 22:10:03.123280 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:03.123258 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6ea374-gbn2r" Oct 30 22:10:03.161439 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:03.161374 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/665fa115-464b-4259-9f81-a2907f42db04-kube-api-access-rccdl\" (UniqueName: \"kubernetes.io/projected/665fa115-464b-4259-9f81-a2907f42db04-kube-api-access-rccdl\") pod \"665fa115-464b-4259-9f81-a2907f42db04\" (UID: \"665fa115-464b-4259-9f81-a2907f42db04\") " Oct 30 22:10:03.161536 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:03.161491 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/665fa115-464b-4259-9f81-a2907f42db04-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/665fa115-464b-4259-9f81-a2907f42db04-maas-api-openshift-ca\") pod \"665fa115-464b-4259-9f81-a2907f42db04\" (UID: \"665fa115-464b-4259-9f81-a2907f42db04\") " Oct 30 22:10:03.161655 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:03.161637 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/665fa115-464b-4259-9f81-a2907f42db04-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/665fa115-464b-4259-9f81-a2907f42db04-maas-api-xks-ca\") pod \"665fa115-464b-4259-9f81-a2907f42db04\" (UID: \"665fa115-464b-4259-9f81-a2907f42db04\") " Oct 30 22:10:03.161873 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:03.161843 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/665fa115-464b-4259-9f81-a2907f42db04-maas-api-openshift-ca" pod "665fa115-464b-4259-9f81-a2907f42db04" (UID: "665fa115-464b-4259-9f81-a2907f42db04"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:10:03.162147 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:03.162126 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/665fa115-464b-4259-9f81-a2907f42db04-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:03.163600 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:03.163576 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/665fa115-464b-4259-9f81-a2907f42db04-kube-api-access-rccdl" pod "665fa115-464b-4259-9f81-a2907f42db04" (UID: "665fa115-464b-4259-9f81-a2907f42db04"). InnerVolumeSpecName "kube-api-access-rccdl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:03.164055 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:03.164038 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/665fa115-464b-4259-9f81-a2907f42db04-maas-api-xks-ca" pod "665fa115-464b-4259-9f81-a2907f42db04" (UID: "665fa115-464b-4259-9f81-a2907f42db04"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:10:03.263448 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:03.263416 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/665fa115-464b-4259-9f81-a2907f42db04-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:03.263448 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:03.263447 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rccdl\" (UniqueName: \"kubernetes.io/projected/665fa115-464b-4259-9f81-a2907f42db04-kube-api-access-rccdl\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:03.902977 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:03.902939 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6ea374-gbn2r" Oct 30 22:10:03.902977 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:03.902954 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6ea374-gbn2r" event={"ID":"665fa115-464b-4259-9f81-a2907f42db04","Type":"ContainerDied","Data":"479589d390e1a17232e01cff8905076129203fb0074a7afd1f7d74d80e205435"} Oct 30 22:10:03.903248 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:03.903015 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="479589d390e1a17232e01cff8905076129203fb0074a7afd1f7d74d80e205435" Oct 30 22:10:04.416803 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:04.416772 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="926fa847-0c3c-43e0-b378-04c82433cd98" path="/var/lib/kubelet/pods/926fa847-0c3c-43e0-b378-04c82433cd98/volumes" Oct 30 22:10:04.598508 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:04.598475 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-6ea374/distinct-2-w1-6ea374-kserve-68fb49889-hpg2k"] Oct 30 22:10:04.649737 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:04.649706 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-6ea374/simulator-w1-6ea374-kserve-775cfdf98b-p5q25"] Oct 30 22:10:10.730262 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:10.730225 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92a1c4d5-c5crj"] Oct 30 22:10:10.730750 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:10.730685 5748 cpu_manager.go:414] "containerMap: removing container" podUID="926fa847-0c3c-43e0-b378-04c82433cd98" containerName="main" Oct 30 22:10:10.730750 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:10.730697 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="926fa847-0c3c-43e0-b378-04c82433cd98" containerName="main" Oct 30 22:10:10.730750 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:10.730725 5748 cpu_manager.go:414] "containerMap: removing container" podUID="dd995578-e1ab-4647-8092-8ddf357b2060" containerName="revoke-keys" Oct 30 22:10:10.730750 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:10.730730 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd995578-e1ab-4647-8092-8ddf357b2060" containerName="revoke-keys" Oct 30 22:10:10.730750 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:10.730745 5748 cpu_manager.go:414] "containerMap: removing container" podUID="665fa115-464b-4259-9f81-a2907f42db04" containerName="revoke-keys" Oct 30 22:10:10.730750 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:10.730750 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="665fa115-464b-4259-9f81-a2907f42db04" containerName="revoke-keys" Oct 30 22:10:10.731026 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:10.730819 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="665fa115-464b-4259-9f81-a2907f42db04" containerName="revoke-keys" Oct 30 22:10:10.731026 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:10.730831 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="926fa847-0c3c-43e0-b378-04c82433cd98" containerName="main" Oct 30 22:10:10.731026 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:10.730841 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd995578-e1ab-4647-8092-8ddf357b2060" containerName="revoke-keys" Oct 30 22:10:10.734011 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:10.733995 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a1c4d5-c5crj" Oct 30 22:10:10.789576 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:10.789543 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:10:10.790597 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:10.790574 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29f06004-7ljm6"] Oct 30 22:10:10.794433 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:10.794409 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f06004-7ljm6" Oct 30 22:10:10.838772 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:10.838743 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:10:10.838941 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:10.838811 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 22:10:10.840081 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:10.840061 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:10:10.842435 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:10.842400 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92a1c4d5-c5crj"] Oct 30 22:10:10.842639 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:10.842619 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88d13e16-48cd-4cbc-8569-c2b46e06f555-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29f06004-7ljm6\" (UID: \"88d13e16-48cd-4cbc-8569-c2b46e06f555\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f06004-7ljm6" Oct 30 22:10:10.842701 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:10.842676 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhtcq\" (UniqueName: \"kubernetes.io/projected/c7e9d383-7fb2-4b0f-ad14-708eb31fb828-kube-api-access-mhtcq\") pod \"maas-api-revoke-sub-92a1c4d5-c5crj\" (UID: \"c7e9d383-7fb2-4b0f-ad14-708eb31fb828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a1c4d5-c5crj" Oct 30 22:10:10.842780 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:10.842763 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7e9d383-7fb2-4b0f-ad14-708eb31fb828-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92a1c4d5-c5crj\" (UID: \"c7e9d383-7fb2-4b0f-ad14-708eb31fb828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a1c4d5-c5crj" Oct 30 22:10:10.842840 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:10.842805 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7e9d383-7fb2-4b0f-ad14-708eb31fb828-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92a1c4d5-c5crj\" (UID: \"c7e9d383-7fb2-4b0f-ad14-708eb31fb828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a1c4d5-c5crj" Oct 30 22:10:10.842926 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:10.842909 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88d13e16-48cd-4cbc-8569-c2b46e06f555-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29f06004-7ljm6\" (UID: \"88d13e16-48cd-4cbc-8569-c2b46e06f555\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f06004-7ljm6" Oct 30 22:10:10.843013 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:10.843000 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7fpw\" (UniqueName: \"kubernetes.io/projected/88d13e16-48cd-4cbc-8569-c2b46e06f555-kube-api-access-w7fpw\") pod \"maas-api-revoke-sub-29f06004-7ljm6\" (UID: \"88d13e16-48cd-4cbc-8569-c2b46e06f555\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f06004-7ljm6" Oct 30 22:10:10.857200 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:10.857172 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29f06004-7ljm6"] Oct 30 22:10:10.943584 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:10.943554 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88d13e16-48cd-4cbc-8569-c2b46e06f555-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29f06004-7ljm6\" (UID: \"88d13e16-48cd-4cbc-8569-c2b46e06f555\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f06004-7ljm6" Oct 30 22:10:10.943748 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:10.943617 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7fpw\" (UniqueName: \"kubernetes.io/projected/88d13e16-48cd-4cbc-8569-c2b46e06f555-kube-api-access-w7fpw\") pod \"maas-api-revoke-sub-29f06004-7ljm6\" (UID: \"88d13e16-48cd-4cbc-8569-c2b46e06f555\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f06004-7ljm6" Oct 30 22:10:10.943748 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:10.943654 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88d13e16-48cd-4cbc-8569-c2b46e06f555-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29f06004-7ljm6\" (UID: \"88d13e16-48cd-4cbc-8569-c2b46e06f555\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f06004-7ljm6" Oct 30 22:10:10.943748 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:10.943722 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhtcq\" (UniqueName: \"kubernetes.io/projected/c7e9d383-7fb2-4b0f-ad14-708eb31fb828-kube-api-access-mhtcq\") pod \"maas-api-revoke-sub-92a1c4d5-c5crj\" (UID: \"c7e9d383-7fb2-4b0f-ad14-708eb31fb828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a1c4d5-c5crj" Oct 30 22:10:10.943748 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:10.943743 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7e9d383-7fb2-4b0f-ad14-708eb31fb828-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92a1c4d5-c5crj\" (UID: \"c7e9d383-7fb2-4b0f-ad14-708eb31fb828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a1c4d5-c5crj" Oct 30 22:10:10.944016 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:10.943771 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7e9d383-7fb2-4b0f-ad14-708eb31fb828-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92a1c4d5-c5crj\" (UID: \"c7e9d383-7fb2-4b0f-ad14-708eb31fb828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a1c4d5-c5crj" Oct 30 22:10:10.944534 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:10.944505 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88d13e16-48cd-4cbc-8569-c2b46e06f555-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29f06004-7ljm6\" (UID: \"88d13e16-48cd-4cbc-8569-c2b46e06f555\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f06004-7ljm6" Oct 30 22:10:10.944654 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:10.944505 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7e9d383-7fb2-4b0f-ad14-708eb31fb828-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92a1c4d5-c5crj\" (UID: \"c7e9d383-7fb2-4b0f-ad14-708eb31fb828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a1c4d5-c5crj" Oct 30 22:10:10.946022 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:10.946000 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88d13e16-48cd-4cbc-8569-c2b46e06f555-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29f06004-7ljm6\" (UID: \"88d13e16-48cd-4cbc-8569-c2b46e06f555\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f06004-7ljm6" Oct 30 22:10:10.946118 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:10.946074 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7e9d383-7fb2-4b0f-ad14-708eb31fb828-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92a1c4d5-c5crj\" (UID: \"c7e9d383-7fb2-4b0f-ad14-708eb31fb828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a1c4d5-c5crj" Oct 30 22:10:11.020864 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:11.020794 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7fpw\" (UniqueName: \"kubernetes.io/projected/88d13e16-48cd-4cbc-8569-c2b46e06f555-kube-api-access-w7fpw\") pod \"maas-api-revoke-sub-29f06004-7ljm6\" (UID: \"88d13e16-48cd-4cbc-8569-c2b46e06f555\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f06004-7ljm6" Oct 30 22:10:11.021009 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:11.020875 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhtcq\" (UniqueName: \"kubernetes.io/projected/c7e9d383-7fb2-4b0f-ad14-708eb31fb828-kube-api-access-mhtcq\") pod \"maas-api-revoke-sub-92a1c4d5-c5crj\" (UID: \"c7e9d383-7fb2-4b0f-ad14-708eb31fb828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a1c4d5-c5crj" Oct 30 22:10:11.048028 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:11.048000 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a1c4d5-c5crj" Oct 30 22:10:11.109492 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:11.109249 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f06004-7ljm6" Oct 30 22:10:11.429099 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:11.429071 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92a1c4d5-c5crj"] Oct 30 22:10:11.431393 ip-10-0-143-61 kubenswrapper[5748]: W1030 22:10:11.431363 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7e9d383_7fb2_4b0f_ad14_708eb31fb828.slice/crio-c6e7154a090a1978c301fc90822d5780eb30e53506d1724e396027baf34c7837 WatchSource:0}: Error finding container c6e7154a090a1978c301fc90822d5780eb30e53506d1724e396027baf34c7837: Status 404 returned error can't find the container with id c6e7154a090a1978c301fc90822d5780eb30e53506d1724e396027baf34c7837 Oct 30 22:10:11.527728 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:11.527702 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29f06004-7ljm6"] Oct 30 22:10:11.530240 ip-10-0-143-61 kubenswrapper[5748]: W1030 22:10:11.530210 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88d13e16_48cd_4cbc_8569_c2b46e06f555.slice/crio-0d0aacd60eb8f6edbf6ba26edfcc77a095152ba4d576f55133dadaacf762b6a1 WatchSource:0}: Error finding container 0d0aacd60eb8f6edbf6ba26edfcc77a095152ba4d576f55133dadaacf762b6a1: Status 404 returned error can't find the container with id 0d0aacd60eb8f6edbf6ba26edfcc77a095152ba4d576f55133dadaacf762b6a1 Oct 30 22:10:11.689313 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:11.689210 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-fd2a94/test-model-fd2a94-kserve-678bdc5f47-h4hxc" podUID="e428337c-c0ce-46b8-9471-96e53e5bc09c" containerName="main" containerID="cri-o://2573c134e47f3366893f064af88bd18c9b842564fbc627cfa6b8e2e99ae3af82" gracePeriod=45 Oct 30 22:10:11.928496 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:11.928470 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fd2a94/test-model-fd2a94-kserve-678bdc5f47-h4hxc" Oct 30 22:10:11.952099 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:11.952007 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a1c4d5-c5crj" event={"ID":"c7e9d383-7fb2-4b0f-ad14-708eb31fb828","Type":"ContainerStarted","Data":"ae7ab12574a9fb859e8a646d6befd78833522f3036badc7ba93e29f790c50ded"} Oct 30 22:10:11.952214 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:11.952108 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a1c4d5-c5crj" event={"ID":"c7e9d383-7fb2-4b0f-ad14-708eb31fb828","Type":"ContainerStarted","Data":"c6e7154a090a1978c301fc90822d5780eb30e53506d1724e396027baf34c7837"} Oct 30 22:10:11.953779 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:11.953737 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f06004-7ljm6" event={"ID":"88d13e16-48cd-4cbc-8569-c2b46e06f555","Type":"ContainerStarted","Data":"8adcc008e518884e05136e75ddba09f51ff8191b5282f65bbd1dc02a30b4e7fa"} Oct 30 22:10:11.953911 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:11.953788 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f06004-7ljm6" event={"ID":"88d13e16-48cd-4cbc-8569-c2b46e06f555","Type":"ContainerStarted","Data":"0d0aacd60eb8f6edbf6ba26edfcc77a095152ba4d576f55133dadaacf762b6a1"} Oct 30 22:10:11.955213 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:11.955183 5748 generic.go:356] "Generic (PLEG): container finished" podID="e428337c-c0ce-46b8-9471-96e53e5bc09c" containerID="2573c134e47f3366893f064af88bd18c9b842564fbc627cfa6b8e2e99ae3af82" exitCode=0 Oct 30 22:10:11.955308 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:11.955215 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fd2a94/test-model-fd2a94-kserve-678bdc5f47-h4hxc" Oct 30 22:10:11.955308 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:11.955216 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fd2a94/test-model-fd2a94-kserve-678bdc5f47-h4hxc" event={"ID":"e428337c-c0ce-46b8-9471-96e53e5bc09c","Type":"ContainerDied","Data":"2573c134e47f3366893f064af88bd18c9b842564fbc627cfa6b8e2e99ae3af82"} Oct 30 22:10:11.955308 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:11.955251 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fd2a94/test-model-fd2a94-kserve-678bdc5f47-h4hxc" event={"ID":"e428337c-c0ce-46b8-9471-96e53e5bc09c","Type":"ContainerDied","Data":"b29b3062e88d0ce1b50f34e16c05887b89f8e98f6c44463573d916bb641bf6e8"} Oct 30 22:10:11.955308 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:11.955267 5748 scope.go:122] "RemoveContainer" containerID="2573c134e47f3366893f064af88bd18c9b842564fbc627cfa6b8e2e99ae3af82" Oct 30 22:10:11.969396 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:11.969381 5748 scope.go:122] "RemoveContainer" containerID="2573c134e47f3366893f064af88bd18c9b842564fbc627cfa6b8e2e99ae3af82" Oct 30 22:10:11.969631 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:10:11.969611 5748 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2573c134e47f3366893f064af88bd18c9b842564fbc627cfa6b8e2e99ae3af82\": container with ID starting with 2573c134e47f3366893f064af88bd18c9b842564fbc627cfa6b8e2e99ae3af82 not found: ID does not exist" containerID="2573c134e47f3366893f064af88bd18c9b842564fbc627cfa6b8e2e99ae3af82" Oct 30 22:10:11.969681 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:11.969646 5748 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2573c134e47f3366893f064af88bd18c9b842564fbc627cfa6b8e2e99ae3af82"} err="failed to get container status \"2573c134e47f3366893f064af88bd18c9b842564fbc627cfa6b8e2e99ae3af82\": rpc error: code = NotFound desc = could not find container \"2573c134e47f3366893f064af88bd18c9b842564fbc627cfa6b8e2e99ae3af82\": container with ID starting with 2573c134e47f3366893f064af88bd18c9b842564fbc627cfa6b8e2e99ae3af82 not found: ID does not exist" Oct 30 22:10:12.059480 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:12.055765 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e428337c-c0ce-46b8-9471-96e53e5bc09c-home\" (UniqueName: \"kubernetes.io/empty-dir/e428337c-c0ce-46b8-9471-96e53e5bc09c-home\") pod \"e428337c-c0ce-46b8-9471-96e53e5bc09c\" (UID: \"e428337c-c0ce-46b8-9471-96e53e5bc09c\") " Oct 30 22:10:12.059480 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:12.055811 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e428337c-c0ce-46b8-9471-96e53e5bc09c-tls-certs\" (UniqueName: \"kubernetes.io/secret/e428337c-c0ce-46b8-9471-96e53e5bc09c-tls-certs\") pod \"e428337c-c0ce-46b8-9471-96e53e5bc09c\" (UID: \"e428337c-c0ce-46b8-9471-96e53e5bc09c\") " Oct 30 22:10:12.059480 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:12.056003 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e428337c-c0ce-46b8-9471-96e53e5bc09c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e428337c-c0ce-46b8-9471-96e53e5bc09c-tmp-dir\") pod \"e428337c-c0ce-46b8-9471-96e53e5bc09c\" (UID: \"e428337c-c0ce-46b8-9471-96e53e5bc09c\") " Oct 30 22:10:12.059480 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:12.056031 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e428337c-c0ce-46b8-9471-96e53e5bc09c-home" pod "e428337c-c0ce-46b8-9471-96e53e5bc09c" (UID: "e428337c-c0ce-46b8-9471-96e53e5bc09c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:12.059480 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:12.056040 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e428337c-c0ce-46b8-9471-96e53e5bc09c-kube-api-access-jm6cl\" (UniqueName: \"kubernetes.io/projected/e428337c-c0ce-46b8-9471-96e53e5bc09c-kube-api-access-jm6cl\") pod \"e428337c-c0ce-46b8-9471-96e53e5bc09c\" (UID: \"e428337c-c0ce-46b8-9471-96e53e5bc09c\") " Oct 30 22:10:12.059480 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:12.056069 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e428337c-c0ce-46b8-9471-96e53e5bc09c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e428337c-c0ce-46b8-9471-96e53e5bc09c-dshm\") pod \"e428337c-c0ce-46b8-9471-96e53e5bc09c\" (UID: \"e428337c-c0ce-46b8-9471-96e53e5bc09c\") " Oct 30 22:10:12.059480 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:12.056200 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e428337c-c0ce-46b8-9471-96e53e5bc09c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e428337c-c0ce-46b8-9471-96e53e5bc09c-model-cache\") pod \"e428337c-c0ce-46b8-9471-96e53e5bc09c\" (UID: \"e428337c-c0ce-46b8-9471-96e53e5bc09c\") " Oct 30 22:10:12.059480 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:12.056594 5748 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e428337c-c0ce-46b8-9471-96e53e5bc09c-home\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:12.059480 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:12.056944 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e428337c-c0ce-46b8-9471-96e53e5bc09c-tmp-dir" pod "e428337c-c0ce-46b8-9471-96e53e5bc09c" (UID: "e428337c-c0ce-46b8-9471-96e53e5bc09c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:12.059480 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:12.057401 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e428337c-c0ce-46b8-9471-96e53e5bc09c-model-cache" pod "e428337c-c0ce-46b8-9471-96e53e5bc09c" (UID: "e428337c-c0ce-46b8-9471-96e53e5bc09c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:12.060266 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:12.060057 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e428337c-c0ce-46b8-9471-96e53e5bc09c-dshm" pod "e428337c-c0ce-46b8-9471-96e53e5bc09c" (UID: "e428337c-c0ce-46b8-9471-96e53e5bc09c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:12.060266 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:12.060071 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e428337c-c0ce-46b8-9471-96e53e5bc09c-tls-certs" pod "e428337c-c0ce-46b8-9471-96e53e5bc09c" (UID: "e428337c-c0ce-46b8-9471-96e53e5bc09c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:10:12.061052 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:12.061022 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e428337c-c0ce-46b8-9471-96e53e5bc09c-kube-api-access-jm6cl" pod "e428337c-c0ce-46b8-9471-96e53e5bc09c" (UID: "e428337c-c0ce-46b8-9471-96e53e5bc09c"). InnerVolumeSpecName "kube-api-access-jm6cl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:12.099221 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:12.099187 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-6ea374-84496c9544-jjw75"] Oct 30 22:10:12.157322 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:12.157288 5748 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e428337c-c0ce-46b8-9471-96e53e5bc09c-tmp-dir\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:12.157322 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:12.157320 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jm6cl\" (UniqueName: \"kubernetes.io/projected/e428337c-c0ce-46b8-9471-96e53e5bc09c-kube-api-access-jm6cl\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:12.157508 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:12.157333 5748 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e428337c-c0ce-46b8-9471-96e53e5bc09c-dshm\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:12.157508 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:12.157345 5748 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e428337c-c0ce-46b8-9471-96e53e5bc09c-model-cache\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:12.157508 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:12.157356 5748 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e428337c-c0ce-46b8-9471-96e53e5bc09c-tls-certs\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:12.212620 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:12.212550 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-6ea374-df875f76c-8bbkw"] Oct 30 22:10:12.221153 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:12.221110 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a1c4d5-c5crj" podStartSLOduration=2.221097125 podStartE2EDuration="2.221097125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:12.220104255 +0000 UTC m=+2795.962625189" watchObservedRunningTime="2026-10-30 22:10:12.221097125 +0000 UTC m=+2795.963618060" Oct 30 22:10:12.336335 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:12.336292 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f06004-7ljm6" podStartSLOduration=2.336278494 podStartE2EDuration="2.336278494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:12.335932176 +0000 UTC m=+2796.078453115" watchObservedRunningTime="2026-10-30 22:10:12.336278494 +0000 UTC m=+2796.078799444" Oct 30 22:10:12.696818 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:12.696782 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-fd2a94/test-model-fd2a94-kserve-678bdc5f47-h4hxc"] Oct 30 22:10:12.700981 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:12.700950 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-fd2a94/test-model-fd2a94-kserve-678bdc5f47-h4hxc"] Oct 30 22:10:12.962715 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:12.962682 5748 generic.go:356] "Generic (PLEG): container finished" podID="88d13e16-48cd-4cbc-8569-c2b46e06f555" containerID="8adcc008e518884e05136e75ddba09f51ff8191b5282f65bbd1dc02a30b4e7fa" exitCode=0 Oct 30 22:10:12.963163 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:12.962771 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f06004-7ljm6" event={"ID":"88d13e16-48cd-4cbc-8569-c2b46e06f555","Type":"ContainerDied","Data":"8adcc008e518884e05136e75ddba09f51ff8191b5282f65bbd1dc02a30b4e7fa"} Oct 30 22:10:12.965239 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:12.965216 5748 generic.go:356] "Generic (PLEG): container finished" podID="c7e9d383-7fb2-4b0f-ad14-708eb31fb828" containerID="ae7ab12574a9fb859e8a646d6befd78833522f3036badc7ba93e29f790c50ded" exitCode=0 Oct 30 22:10:12.965355 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:12.965282 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a1c4d5-c5crj" event={"ID":"c7e9d383-7fb2-4b0f-ad14-708eb31fb828","Type":"ContainerDied","Data":"ae7ab12574a9fb859e8a646d6befd78833522f3036badc7ba93e29f790c50ded"} Oct 30 22:10:14.128599 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:14.128559 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f06004-7ljm6" Oct 30 22:10:14.171775 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:14.171753 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a1c4d5-c5crj" Oct 30 22:10:14.177999 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:14.177979 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88d13e16-48cd-4cbc-8569-c2b46e06f555-kube-api-access-w7fpw\" (UniqueName: \"kubernetes.io/projected/88d13e16-48cd-4cbc-8569-c2b46e06f555-kube-api-access-w7fpw\") pod \"88d13e16-48cd-4cbc-8569-c2b46e06f555\" (UID: \"88d13e16-48cd-4cbc-8569-c2b46e06f555\") " Oct 30 22:10:14.178082 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:14.178026 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88d13e16-48cd-4cbc-8569-c2b46e06f555-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88d13e16-48cd-4cbc-8569-c2b46e06f555-maas-api-xks-ca\") pod \"88d13e16-48cd-4cbc-8569-c2b46e06f555\" (UID: \"88d13e16-48cd-4cbc-8569-c2b46e06f555\") " Oct 30 22:10:14.178165 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:14.178150 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88d13e16-48cd-4cbc-8569-c2b46e06f555-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88d13e16-48cd-4cbc-8569-c2b46e06f555-maas-api-openshift-ca\") pod \"88d13e16-48cd-4cbc-8569-c2b46e06f555\" (UID: \"88d13e16-48cd-4cbc-8569-c2b46e06f555\") " Oct 30 22:10:14.178498 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:14.178475 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88d13e16-48cd-4cbc-8569-c2b46e06f555-maas-api-openshift-ca" pod "88d13e16-48cd-4cbc-8569-c2b46e06f555" (UID: "88d13e16-48cd-4cbc-8569-c2b46e06f555"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:10:14.180204 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:14.180184 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88d13e16-48cd-4cbc-8569-c2b46e06f555-maas-api-xks-ca" pod "88d13e16-48cd-4cbc-8569-c2b46e06f555" (UID: "88d13e16-48cd-4cbc-8569-c2b46e06f555"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:10:14.180292 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:14.180239 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88d13e16-48cd-4cbc-8569-c2b46e06f555-kube-api-access-w7fpw" pod "88d13e16-48cd-4cbc-8569-c2b46e06f555" (UID: "88d13e16-48cd-4cbc-8569-c2b46e06f555"). InnerVolumeSpecName "kube-api-access-w7fpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:14.279761 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:14.279658 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7e9d383-7fb2-4b0f-ad14-708eb31fb828-kube-api-access-mhtcq\" (UniqueName: \"kubernetes.io/projected/c7e9d383-7fb2-4b0f-ad14-708eb31fb828-kube-api-access-mhtcq\") pod \"c7e9d383-7fb2-4b0f-ad14-708eb31fb828\" (UID: \"c7e9d383-7fb2-4b0f-ad14-708eb31fb828\") " Oct 30 22:10:14.279761 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:14.279737 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7e9d383-7fb2-4b0f-ad14-708eb31fb828-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7e9d383-7fb2-4b0f-ad14-708eb31fb828-maas-api-xks-ca\") pod \"c7e9d383-7fb2-4b0f-ad14-708eb31fb828\" (UID: \"c7e9d383-7fb2-4b0f-ad14-708eb31fb828\") " Oct 30 22:10:14.280040 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:14.279945 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7e9d383-7fb2-4b0f-ad14-708eb31fb828-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7e9d383-7fb2-4b0f-ad14-708eb31fb828-maas-api-openshift-ca\") pod \"c7e9d383-7fb2-4b0f-ad14-708eb31fb828\" (UID: \"c7e9d383-7fb2-4b0f-ad14-708eb31fb828\") " Oct 30 22:10:14.280590 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:14.280392 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88d13e16-48cd-4cbc-8569-c2b46e06f555-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:14.280590 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:14.280421 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7fpw\" (UniqueName: \"kubernetes.io/projected/88d13e16-48cd-4cbc-8569-c2b46e06f555-kube-api-access-w7fpw\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:14.280590 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:14.280437 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88d13e16-48cd-4cbc-8569-c2b46e06f555-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:14.280590 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:14.280535 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7e9d383-7fb2-4b0f-ad14-708eb31fb828-maas-api-openshift-ca" pod "c7e9d383-7fb2-4b0f-ad14-708eb31fb828" (UID: "c7e9d383-7fb2-4b0f-ad14-708eb31fb828"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:10:14.282407 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:14.282378 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7e9d383-7fb2-4b0f-ad14-708eb31fb828-maas-api-xks-ca" pod "c7e9d383-7fb2-4b0f-ad14-708eb31fb828" (UID: "c7e9d383-7fb2-4b0f-ad14-708eb31fb828"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:10:14.282805 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:14.282787 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7e9d383-7fb2-4b0f-ad14-708eb31fb828-kube-api-access-mhtcq" pod "c7e9d383-7fb2-4b0f-ad14-708eb31fb828" (UID: "c7e9d383-7fb2-4b0f-ad14-708eb31fb828"). InnerVolumeSpecName "kube-api-access-mhtcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:14.381072 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:14.381038 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7e9d383-7fb2-4b0f-ad14-708eb31fb828-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:14.381072 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:14.381066 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhtcq\" (UniqueName: \"kubernetes.io/projected/c7e9d383-7fb2-4b0f-ad14-708eb31fb828-kube-api-access-mhtcq\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:14.381072 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:14.381077 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7e9d383-7fb2-4b0f-ad14-708eb31fb828-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:14.418240 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:14.418209 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e428337c-c0ce-46b8-9471-96e53e5bc09c" path="/var/lib/kubelet/pods/e428337c-c0ce-46b8-9471-96e53e5bc09c/volumes" Oct 30 22:10:14.978229 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:14.978199 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f06004-7ljm6" Oct 30 22:10:14.978421 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:14.978230 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f06004-7ljm6" event={"ID":"88d13e16-48cd-4cbc-8569-c2b46e06f555","Type":"ContainerDied","Data":"0d0aacd60eb8f6edbf6ba26edfcc77a095152ba4d576f55133dadaacf762b6a1"} Oct 30 22:10:14.978421 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:14.978281 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d0aacd60eb8f6edbf6ba26edfcc77a095152ba4d576f55133dadaacf762b6a1" Oct 30 22:10:14.979793 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:14.979770 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a1c4d5-c5crj" Oct 30 22:10:14.979793 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:14.979788 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a1c4d5-c5crj" event={"ID":"c7e9d383-7fb2-4b0f-ad14-708eb31fb828","Type":"ContainerDied","Data":"c6e7154a090a1978c301fc90822d5780eb30e53506d1724e396027baf34c7837"} Oct 30 22:10:14.980002 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:14.979826 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6e7154a090a1978c301fc90822d5780eb30e53506d1724e396027baf34c7837" Oct 30 22:10:15.497356 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:15.497322 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ea1f0b/test-model-ea1f0b-kserve-586698c74f-dmv5g"] Oct 30 22:10:16.117020 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:16.116985 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd2a94-wgrgm"] Oct 30 22:10:16.117557 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:16.117537 5748 cpu_manager.go:414] "containerMap: removing container" podUID="88d13e16-48cd-4cbc-8569-c2b46e06f555" containerName="revoke-keys" Oct 30 22:10:16.117557 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:16.117558 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88d13e16-48cd-4cbc-8569-c2b46e06f555" containerName="revoke-keys" Oct 30 22:10:16.117738 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:16.117576 5748 cpu_manager.go:414] "containerMap: removing container" podUID="e428337c-c0ce-46b8-9471-96e53e5bc09c" containerName="main" Oct 30 22:10:16.117738 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:16.117583 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e428337c-c0ce-46b8-9471-96e53e5bc09c" containerName="main" Oct 30 22:10:16.117738 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:16.117612 5748 cpu_manager.go:414] "containerMap: removing container" podUID="c7e9d383-7fb2-4b0f-ad14-708eb31fb828" containerName="revoke-keys" Oct 30 22:10:16.117738 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:16.117619 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7e9d383-7fb2-4b0f-ad14-708eb31fb828" containerName="revoke-keys" Oct 30 22:10:16.117953 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:16.117747 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="88d13e16-48cd-4cbc-8569-c2b46e06f555" containerName="revoke-keys" Oct 30 22:10:16.117953 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:16.117762 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="e428337c-c0ce-46b8-9471-96e53e5bc09c" containerName="main" Oct 30 22:10:16.117953 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:16.117777 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7e9d383-7fb2-4b0f-ad14-708eb31fb828" containerName="revoke-keys" Oct 30 22:10:16.122223 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:16.122203 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd2a94-wgrgm" Oct 30 22:10:16.176991 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:16.176960 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:10:16.177292 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:16.177265 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:10:16.180834 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:16.180815 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd2a94-wgrgm"] Oct 30 22:10:16.201490 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:16.201459 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8zb5\" (UniqueName: \"kubernetes.io/projected/ded0f467-f8cb-4b8e-89e8-fbf29cc3189f-kube-api-access-z8zb5\") pod \"maas-api-revoke-keys-e2e-inf-a-fd2a94-wgrgm\" (UID: \"ded0f467-f8cb-4b8e-89e8-fbf29cc3189f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd2a94-wgrgm" Oct 30 22:10:16.201633 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:16.201507 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ded0f467-f8cb-4b8e-89e8-fbf29cc3189f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fd2a94-wgrgm\" (UID: \"ded0f467-f8cb-4b8e-89e8-fbf29cc3189f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd2a94-wgrgm" Oct 30 22:10:16.201633 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:16.201594 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ded0f467-f8cb-4b8e-89e8-fbf29cc3189f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fd2a94-wgrgm\" (UID: \"ded0f467-f8cb-4b8e-89e8-fbf29cc3189f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd2a94-wgrgm" Oct 30 22:10:16.212038 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:16.212018 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:10:16.212155 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:16.212100 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 22:10:16.303196 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:16.303158 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8zb5\" (UniqueName: \"kubernetes.io/projected/ded0f467-f8cb-4b8e-89e8-fbf29cc3189f-kube-api-access-z8zb5\") pod \"maas-api-revoke-keys-e2e-inf-a-fd2a94-wgrgm\" (UID: \"ded0f467-f8cb-4b8e-89e8-fbf29cc3189f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd2a94-wgrgm" Oct 30 22:10:16.303363 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:16.303209 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ded0f467-f8cb-4b8e-89e8-fbf29cc3189f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fd2a94-wgrgm\" (UID: \"ded0f467-f8cb-4b8e-89e8-fbf29cc3189f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd2a94-wgrgm" Oct 30 22:10:16.303363 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:16.303236 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ded0f467-f8cb-4b8e-89e8-fbf29cc3189f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fd2a94-wgrgm\" (UID: \"ded0f467-f8cb-4b8e-89e8-fbf29cc3189f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd2a94-wgrgm" Oct 30 22:10:16.304016 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:16.303994 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ded0f467-f8cb-4b8e-89e8-fbf29cc3189f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fd2a94-wgrgm\" (UID: \"ded0f467-f8cb-4b8e-89e8-fbf29cc3189f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd2a94-wgrgm" Oct 30 22:10:16.305547 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:16.305525 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ded0f467-f8cb-4b8e-89e8-fbf29cc3189f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fd2a94-wgrgm\" (UID: \"ded0f467-f8cb-4b8e-89e8-fbf29cc3189f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd2a94-wgrgm" Oct 30 22:10:16.364710 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:16.364680 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8zb5\" (UniqueName: \"kubernetes.io/projected/ded0f467-f8cb-4b8e-89e8-fbf29cc3189f-kube-api-access-z8zb5\") pod \"maas-api-revoke-keys-e2e-inf-a-fd2a94-wgrgm\" (UID: \"ded0f467-f8cb-4b8e-89e8-fbf29cc3189f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd2a94-wgrgm" Oct 30 22:10:16.436289 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:16.436257 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd2a94-wgrgm" Oct 30 22:10:16.814378 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:16.814339 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd2a94-wgrgm"] Oct 30 22:10:16.817410 ip-10-0-143-61 kubenswrapper[5748]: W1030 22:10:16.817382 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podded0f467_f8cb_4b8e_89e8_fbf29cc3189f.slice/crio-218fbe0b48257101ee6239b954ce2051c2d67dac59db9a6e574571c5aaa44798 WatchSource:0}: Error finding container 218fbe0b48257101ee6239b954ce2051c2d67dac59db9a6e574571c5aaa44798: Status 404 returned error can't find the container with id 218fbe0b48257101ee6239b954ce2051c2d67dac59db9a6e574571c5aaa44798 Oct 30 22:10:16.994328 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:16.994286 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd2a94-wgrgm" event={"ID":"ded0f467-f8cb-4b8e-89e8-fbf29cc3189f","Type":"ContainerStarted","Data":"218fbe0b48257101ee6239b954ce2051c2d67dac59db9a6e574571c5aaa44798"} Oct 30 22:10:17.100291 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:17.100188 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-6ea374-84496c9544-jjw75" podUID="eed8b35c-e416-4173-98a7-ad7b3d6bc0ea" containerName="payload-processing" containerID="cri-o://53f3a622dd741bc5719560eedbef8f8c45e92a74d6de112144dd104e7270ee6a" gracePeriod=25 Oct 30 22:10:17.213093 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:17.213057 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6ea374-df875f76c-8bbkw" podUID="084e23c1-9a28-4b27-ab32-d3c348fd2c1b" containerName="payload-pre-processing" containerID="cri-o://0d2b4c6501d7111d9d03377c4aca0ace6a4ba091efd1ca3b803b5155e4e96c43" gracePeriod=25 Oct 30 22:10:17.371690 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:17.371620 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-6ea374-84496c9544-jjw75" Oct 30 22:10:17.415356 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:17.415319 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eed8b35c-e416-4173-98a7-ad7b3d6bc0ea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eed8b35c-e416-4173-98a7-ad7b3d6bc0ea-plugins-config-volume\") pod \"eed8b35c-e416-4173-98a7-ad7b3d6bc0ea\" (UID: \"eed8b35c-e416-4173-98a7-ad7b3d6bc0ea\") " Oct 30 22:10:17.415535 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:17.415447 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eed8b35c-e416-4173-98a7-ad7b3d6bc0ea-kube-api-access-f8mxn\" (UniqueName: \"kubernetes.io/projected/eed8b35c-e416-4173-98a7-ad7b3d6bc0ea-kube-api-access-f8mxn\") pod \"eed8b35c-e416-4173-98a7-ad7b3d6bc0ea\" (UID: \"eed8b35c-e416-4173-98a7-ad7b3d6bc0ea\") " Oct 30 22:10:17.416547 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:17.416514 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eed8b35c-e416-4173-98a7-ad7b3d6bc0ea-plugins-config-volume" pod "eed8b35c-e416-4173-98a7-ad7b3d6bc0ea" (UID: "eed8b35c-e416-4173-98a7-ad7b3d6bc0ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:10:17.419195 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:17.419164 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eed8b35c-e416-4173-98a7-ad7b3d6bc0ea-kube-api-access-f8mxn" pod "eed8b35c-e416-4173-98a7-ad7b3d6bc0ea" (UID: "eed8b35c-e416-4173-98a7-ad7b3d6bc0ea"). InnerVolumeSpecName "kube-api-access-f8mxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:17.480996 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:17.480970 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6ea374-df875f76c-8bbkw" Oct 30 22:10:17.516777 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:17.516741 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/084e23c1-9a28-4b27-ab32-d3c348fd2c1b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/084e23c1-9a28-4b27-ab32-d3c348fd2c1b-plugins-config-volume\") pod \"084e23c1-9a28-4b27-ab32-d3c348fd2c1b\" (UID: \"084e23c1-9a28-4b27-ab32-d3c348fd2c1b\") " Oct 30 22:10:17.516963 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:17.516837 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/084e23c1-9a28-4b27-ab32-d3c348fd2c1b-kube-api-access-jvfh8\" (UniqueName: \"kubernetes.io/projected/084e23c1-9a28-4b27-ab32-d3c348fd2c1b-kube-api-access-jvfh8\") pod \"084e23c1-9a28-4b27-ab32-d3c348fd2c1b\" (UID: \"084e23c1-9a28-4b27-ab32-d3c348fd2c1b\") " Oct 30 22:10:17.517192 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:17.517166 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/084e23c1-9a28-4b27-ab32-d3c348fd2c1b-plugins-config-volume" pod "084e23c1-9a28-4b27-ab32-d3c348fd2c1b" (UID: "084e23c1-9a28-4b27-ab32-d3c348fd2c1b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:10:17.517283 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:17.517194 5748 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eed8b35c-e416-4173-98a7-ad7b3d6bc0ea-plugins-config-volume\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:17.517283 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:17.517207 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8mxn\" (UniqueName: \"kubernetes.io/projected/eed8b35c-e416-4173-98a7-ad7b3d6bc0ea-kube-api-access-f8mxn\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:17.518938 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:17.518890 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/084e23c1-9a28-4b27-ab32-d3c348fd2c1b-kube-api-access-jvfh8" pod "084e23c1-9a28-4b27-ab32-d3c348fd2c1b" (UID: "084e23c1-9a28-4b27-ab32-d3c348fd2c1b"). InnerVolumeSpecName "kube-api-access-jvfh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:17.618812 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:17.618771 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvfh8\" (UniqueName: \"kubernetes.io/projected/084e23c1-9a28-4b27-ab32-d3c348fd2c1b-kube-api-access-jvfh8\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:17.618812 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:17.618811 5748 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/084e23c1-9a28-4b27-ab32-d3c348fd2c1b-plugins-config-volume\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:18.002464 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:18.002431 5748 generic.go:356] "Generic (PLEG): container finished" podID="eed8b35c-e416-4173-98a7-ad7b3d6bc0ea" containerID="53f3a622dd741bc5719560eedbef8f8c45e92a74d6de112144dd104e7270ee6a" exitCode=0 Oct 30 22:10:18.002994 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:18.002473 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-6ea374-84496c9544-jjw75" Oct 30 22:10:18.002994 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:18.002522 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-6ea374-84496c9544-jjw75" event={"ID":"eed8b35c-e416-4173-98a7-ad7b3d6bc0ea","Type":"ContainerDied","Data":"53f3a622dd741bc5719560eedbef8f8c45e92a74d6de112144dd104e7270ee6a"} Oct 30 22:10:18.002994 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:18.002575 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-6ea374-84496c9544-jjw75" event={"ID":"eed8b35c-e416-4173-98a7-ad7b3d6bc0ea","Type":"ContainerDied","Data":"65e300893b541cd56bc5fa57fd193eeb304bc7134de8a01ae811c8e317cc4d88"} Oct 30 22:10:18.002994 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:18.002603 5748 scope.go:122] "RemoveContainer" containerID="53f3a622dd741bc5719560eedbef8f8c45e92a74d6de112144dd104e7270ee6a" Oct 30 22:10:18.004247 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:18.004223 5748 generic.go:356] "Generic (PLEG): container finished" podID="ded0f467-f8cb-4b8e-89e8-fbf29cc3189f" containerID="b3bfa28688f5dd4c624e37c066392f56f4ca0994fc7b9452ac836bbc2339901d" exitCode=0 Oct 30 22:10:18.004354 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:18.004260 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd2a94-wgrgm" event={"ID":"ded0f467-f8cb-4b8e-89e8-fbf29cc3189f","Type":"ContainerDied","Data":"b3bfa28688f5dd4c624e37c066392f56f4ca0994fc7b9452ac836bbc2339901d"} Oct 30 22:10:18.005835 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:18.005817 5748 generic.go:356] "Generic (PLEG): container finished" podID="084e23c1-9a28-4b27-ab32-d3c348fd2c1b" containerID="0d2b4c6501d7111d9d03377c4aca0ace6a4ba091efd1ca3b803b5155e4e96c43" exitCode=0 Oct 30 22:10:18.005962 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:18.005855 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6ea374-df875f76c-8bbkw" Oct 30 22:10:18.005962 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:18.005912 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6ea374-df875f76c-8bbkw" event={"ID":"084e23c1-9a28-4b27-ab32-d3c348fd2c1b","Type":"ContainerDied","Data":"0d2b4c6501d7111d9d03377c4aca0ace6a4ba091efd1ca3b803b5155e4e96c43"} Oct 30 22:10:18.006045 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:18.005963 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6ea374-df875f76c-8bbkw" event={"ID":"084e23c1-9a28-4b27-ab32-d3c348fd2c1b","Type":"ContainerDied","Data":"56bb3285b757dba2b4fef354747ace90d2bedd91dd51743ef9b6164c1282e3a3"} Oct 30 22:10:18.013106 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:18.013088 5748 scope.go:122] "RemoveContainer" containerID="53f3a622dd741bc5719560eedbef8f8c45e92a74d6de112144dd104e7270ee6a" Oct 30 22:10:18.013409 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:10:18.013391 5748 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53f3a622dd741bc5719560eedbef8f8c45e92a74d6de112144dd104e7270ee6a\": container with ID starting with 53f3a622dd741bc5719560eedbef8f8c45e92a74d6de112144dd104e7270ee6a not found: ID does not exist" containerID="53f3a622dd741bc5719560eedbef8f8c45e92a74d6de112144dd104e7270ee6a" Oct 30 22:10:18.013507 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:18.013419 5748 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53f3a622dd741bc5719560eedbef8f8c45e92a74d6de112144dd104e7270ee6a"} err="failed to get container status \"53f3a622dd741bc5719560eedbef8f8c45e92a74d6de112144dd104e7270ee6a\": rpc error: code = NotFound desc = could not find container \"53f3a622dd741bc5719560eedbef8f8c45e92a74d6de112144dd104e7270ee6a\": container with ID starting with 53f3a622dd741bc5719560eedbef8f8c45e92a74d6de112144dd104e7270ee6a not found: ID does not exist" Oct 30 22:10:18.013507 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:18.013441 5748 scope.go:122] "RemoveContainer" containerID="0d2b4c6501d7111d9d03377c4aca0ace6a4ba091efd1ca3b803b5155e4e96c43" Oct 30 22:10:18.022116 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:18.022098 5748 scope.go:122] "RemoveContainer" containerID="0d2b4c6501d7111d9d03377c4aca0ace6a4ba091efd1ca3b803b5155e4e96c43" Oct 30 22:10:18.022360 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:10:18.022341 5748 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d2b4c6501d7111d9d03377c4aca0ace6a4ba091efd1ca3b803b5155e4e96c43\": container with ID starting with 0d2b4c6501d7111d9d03377c4aca0ace6a4ba091efd1ca3b803b5155e4e96c43 not found: ID does not exist" containerID="0d2b4c6501d7111d9d03377c4aca0ace6a4ba091efd1ca3b803b5155e4e96c43" Oct 30 22:10:18.022427 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:18.022366 5748 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d2b4c6501d7111d9d03377c4aca0ace6a4ba091efd1ca3b803b5155e4e96c43"} err="failed to get container status \"0d2b4c6501d7111d9d03377c4aca0ace6a4ba091efd1ca3b803b5155e4e96c43\": rpc error: code = NotFound desc = could not find container \"0d2b4c6501d7111d9d03377c4aca0ace6a4ba091efd1ca3b803b5155e4e96c43\": container with ID starting with 0d2b4c6501d7111d9d03377c4aca0ace6a4ba091efd1ca3b803b5155e4e96c43 not found: ID does not exist" Oct 30 22:10:18.170466 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:18.170433 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-6ea374-84496c9544-jjw75"] Oct 30 22:10:18.220515 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:18.220486 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-6ea374-84496c9544-jjw75"] Oct 30 22:10:18.401300 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:18.401266 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-6ea374-df875f76c-8bbkw"] Oct 30 22:10:18.409047 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:18.409022 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-6ea374-df875f76c-8bbkw"] Oct 30 22:10:18.416367 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:18.416340 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="084e23c1-9a28-4b27-ab32-d3c348fd2c1b" path="/var/lib/kubelet/pods/084e23c1-9a28-4b27-ab32-d3c348fd2c1b/volumes" Oct 30 22:10:18.416738 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:18.416723 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eed8b35c-e416-4173-98a7-ad7b3d6bc0ea" path="/var/lib/kubelet/pods/eed8b35c-e416-4173-98a7-ad7b3d6bc0ea/volumes" Oct 30 22:10:19.154727 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:19.154702 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd2a94-wgrgm" Oct 30 22:10:19.235448 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:19.235418 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ded0f467-f8cb-4b8e-89e8-fbf29cc3189f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ded0f467-f8cb-4b8e-89e8-fbf29cc3189f-maas-api-openshift-ca\") pod \"ded0f467-f8cb-4b8e-89e8-fbf29cc3189f\" (UID: \"ded0f467-f8cb-4b8e-89e8-fbf29cc3189f\") " Oct 30 22:10:19.235638 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:19.235483 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ded0f467-f8cb-4b8e-89e8-fbf29cc3189f-kube-api-access-z8zb5\" (UniqueName: \"kubernetes.io/projected/ded0f467-f8cb-4b8e-89e8-fbf29cc3189f-kube-api-access-z8zb5\") pod \"ded0f467-f8cb-4b8e-89e8-fbf29cc3189f\" (UID: \"ded0f467-f8cb-4b8e-89e8-fbf29cc3189f\") " Oct 30 22:10:19.235638 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:19.235583 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ded0f467-f8cb-4b8e-89e8-fbf29cc3189f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ded0f467-f8cb-4b8e-89e8-fbf29cc3189f-maas-api-xks-ca\") pod \"ded0f467-f8cb-4b8e-89e8-fbf29cc3189f\" (UID: \"ded0f467-f8cb-4b8e-89e8-fbf29cc3189f\") " Oct 30 22:10:19.235883 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:19.235855 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ded0f467-f8cb-4b8e-89e8-fbf29cc3189f-maas-api-openshift-ca" pod "ded0f467-f8cb-4b8e-89e8-fbf29cc3189f" (UID: "ded0f467-f8cb-4b8e-89e8-fbf29cc3189f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:10:19.237683 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:19.237652 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ded0f467-f8cb-4b8e-89e8-fbf29cc3189f-kube-api-access-z8zb5" pod "ded0f467-f8cb-4b8e-89e8-fbf29cc3189f" (UID: "ded0f467-f8cb-4b8e-89e8-fbf29cc3189f"). InnerVolumeSpecName "kube-api-access-z8zb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:19.237812 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:19.237753 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ded0f467-f8cb-4b8e-89e8-fbf29cc3189f-maas-api-xks-ca" pod "ded0f467-f8cb-4b8e-89e8-fbf29cc3189f" (UID: "ded0f467-f8cb-4b8e-89e8-fbf29cc3189f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:10:19.339856 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:19.339769 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ded0f467-f8cb-4b8e-89e8-fbf29cc3189f-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:19.339856 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:19.339803 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ded0f467-f8cb-4b8e-89e8-fbf29cc3189f-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:19.339856 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:19.339818 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8zb5\" (UniqueName: \"kubernetes.io/projected/ded0f467-f8cb-4b8e-89e8-fbf29cc3189f-kube-api-access-z8zb5\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:19.618956 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:19.618815 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-6ea374/distinct-2-w1-6ea374-kserve-68fb49889-hpg2k" podUID="ab533734-e0d8-4acb-854d-12851a87bc58" containerName="main" containerID="cri-o://74b28c950466bf3af4dedf9b1a99435428282a5233f0320db13e4042448e598f" gracePeriod=45 Oct 30 22:10:19.670942 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:19.670856 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-6ea374/simulator-w1-6ea374-kserve-775cfdf98b-p5q25" podUID="b0d53207-65f3-4d43-baba-66bc4c366885" containerName="main" containerID="cri-o://d426b45d403be493a1b773daefab3470ebd05ab0be1be9a6b4577e4bdcc91a73" gracePeriod=45 Oct 30 22:10:19.776537 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:19.776255 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ea374/distinct-2-w1-6ea374-kserve-68fb49889-hpg2k" podUID="ab533734-e0d8-4acb-854d-12851a87bc58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": dial tcp 10.134.0.83:8000: connect: connection refused" Oct 30 22:10:19.941982 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:19.941947 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6ea374/distinct-2-w1-6ea374-kserve-68fb49889-hpg2k" Oct 30 22:10:19.992569 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:19.992538 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6ea374/simulator-w1-6ea374-kserve-775cfdf98b-p5q25" Oct 30 22:10:20.020490 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:20.020460 5748 generic.go:356] "Generic (PLEG): container finished" podID="b0d53207-65f3-4d43-baba-66bc4c366885" containerID="d426b45d403be493a1b773daefab3470ebd05ab0be1be9a6b4577e4bdcc91a73" exitCode=0 Oct 30 22:10:20.020659 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:20.020510 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6ea374/simulator-w1-6ea374-kserve-775cfdf98b-p5q25" Oct 30 22:10:20.020659 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:20.020548 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6ea374/simulator-w1-6ea374-kserve-775cfdf98b-p5q25" event={"ID":"b0d53207-65f3-4d43-baba-66bc4c366885","Type":"ContainerDied","Data":"d426b45d403be493a1b773daefab3470ebd05ab0be1be9a6b4577e4bdcc91a73"} Oct 30 22:10:20.020659 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:20.020601 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6ea374/simulator-w1-6ea374-kserve-775cfdf98b-p5q25" event={"ID":"b0d53207-65f3-4d43-baba-66bc4c366885","Type":"ContainerDied","Data":"a114c8a654a51aae66357f3f87ebe2e0ce50f9f3e1ccd24d56d5cd0f813af11c"} Oct 30 22:10:20.020659 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:20.020624 5748 scope.go:122] "RemoveContainer" containerID="d426b45d403be493a1b773daefab3470ebd05ab0be1be9a6b4577e4bdcc91a73" Oct 30 22:10:20.022130 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:20.022114 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd2a94-wgrgm" Oct 30 22:10:20.022204 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:20.022125 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd2a94-wgrgm" event={"ID":"ded0f467-f8cb-4b8e-89e8-fbf29cc3189f","Type":"ContainerDied","Data":"218fbe0b48257101ee6239b954ce2051c2d67dac59db9a6e574571c5aaa44798"} Oct 30 22:10:20.022204 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:20.022188 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="218fbe0b48257101ee6239b954ce2051c2d67dac59db9a6e574571c5aaa44798" Oct 30 22:10:20.023696 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:20.023669 5748 generic.go:356] "Generic (PLEG): container finished" podID="ab533734-e0d8-4acb-854d-12851a87bc58" containerID="74b28c950466bf3af4dedf9b1a99435428282a5233f0320db13e4042448e598f" exitCode=0 Oct 30 22:10:20.023816 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:20.023721 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6ea374/distinct-2-w1-6ea374-kserve-68fb49889-hpg2k" Oct 30 22:10:20.023816 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:20.023731 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6ea374/distinct-2-w1-6ea374-kserve-68fb49889-hpg2k" event={"ID":"ab533734-e0d8-4acb-854d-12851a87bc58","Type":"ContainerDied","Data":"74b28c950466bf3af4dedf9b1a99435428282a5233f0320db13e4042448e598f"} Oct 30 22:10:20.023816 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:20.023763 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6ea374/distinct-2-w1-6ea374-kserve-68fb49889-hpg2k" event={"ID":"ab533734-e0d8-4acb-854d-12851a87bc58","Type":"ContainerDied","Data":"1ac38ee86ae98400b1272c3aa7869d630632381dd7b633ed43117d4ce80204ed"} Oct 30 22:10:20.032051 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:20.032030 5748 scope.go:122] "RemoveContainer" containerID="d426b45d403be493a1b773daefab3470ebd05ab0be1be9a6b4577e4bdcc91a73" Oct 30 22:10:20.032374 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:10:20.032346 5748 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d426b45d403be493a1b773daefab3470ebd05ab0be1be9a6b4577e4bdcc91a73\": container with ID starting with d426b45d403be493a1b773daefab3470ebd05ab0be1be9a6b4577e4bdcc91a73 not found: ID does not exist" containerID="d426b45d403be493a1b773daefab3470ebd05ab0be1be9a6b4577e4bdcc91a73" Oct 30 22:10:20.032454 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:20.032387 5748 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d426b45d403be493a1b773daefab3470ebd05ab0be1be9a6b4577e4bdcc91a73"} err="failed to get container status \"d426b45d403be493a1b773daefab3470ebd05ab0be1be9a6b4577e4bdcc91a73\": rpc error: code = NotFound desc = could not find container \"d426b45d403be493a1b773daefab3470ebd05ab0be1be9a6b4577e4bdcc91a73\": container with ID starting with d426b45d403be493a1b773daefab3470ebd05ab0be1be9a6b4577e4bdcc91a73 not found: ID does not exist" Oct 30 22:10:20.032454 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:20.032412 5748 scope.go:122] "RemoveContainer" containerID="74b28c950466bf3af4dedf9b1a99435428282a5233f0320db13e4042448e598f" Oct 30 22:10:20.043186 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:20.043163 5748 scope.go:122] "RemoveContainer" containerID="74b28c950466bf3af4dedf9b1a99435428282a5233f0320db13e4042448e598f" Oct 30 22:10:20.043462 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:10:20.043444 5748 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74b28c950466bf3af4dedf9b1a99435428282a5233f0320db13e4042448e598f\": container with ID starting with 74b28c950466bf3af4dedf9b1a99435428282a5233f0320db13e4042448e598f not found: ID does not exist" containerID="74b28c950466bf3af4dedf9b1a99435428282a5233f0320db13e4042448e598f" Oct 30 22:10:20.043525 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:20.043468 5748 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74b28c950466bf3af4dedf9b1a99435428282a5233f0320db13e4042448e598f"} err="failed to get container status \"74b28c950466bf3af4dedf9b1a99435428282a5233f0320db13e4042448e598f\": rpc error: code = NotFound desc = could not find container \"74b28c950466bf3af4dedf9b1a99435428282a5233f0320db13e4042448e598f\": container with ID starting with 74b28c950466bf3af4dedf9b1a99435428282a5233f0320db13e4042448e598f not found: ID does not exist" Oct 30 22:10:20.051766 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:20.051744 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0d53207-65f3-4d43-baba-66bc4c366885-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0d53207-65f3-4d43-baba-66bc4c366885-model-cache\") pod \"b0d53207-65f3-4d43-baba-66bc4c366885\" (UID: \"b0d53207-65f3-4d43-baba-66bc4c366885\") " Oct 30 22:10:20.051865 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:20.051780 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab533734-e0d8-4acb-854d-12851a87bc58-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab533734-e0d8-4acb-854d-12851a87bc58-tmp-dir\") pod \"ab533734-e0d8-4acb-854d-12851a87bc58\" (UID: \"ab533734-e0d8-4acb-854d-12851a87bc58\") " Oct 30 22:10:20.051865 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:20.051810 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0d53207-65f3-4d43-baba-66bc4c366885-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0d53207-65f3-4d43-baba-66bc4c366885-dshm\") pod \"b0d53207-65f3-4d43-baba-66bc4c366885\" (UID: \"b0d53207-65f3-4d43-baba-66bc4c366885\") " Oct 30 22:10:20.051865 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:20.051860 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab533734-e0d8-4acb-854d-12851a87bc58-kube-api-access-8jdf8\" (UniqueName: \"kubernetes.io/projected/ab533734-e0d8-4acb-854d-12851a87bc58-kube-api-access-8jdf8\") pod \"ab533734-e0d8-4acb-854d-12851a87bc58\" (UID: \"ab533734-e0d8-4acb-854d-12851a87bc58\") " Oct 30 22:10:20.052052 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:20.051926 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0d53207-65f3-4d43-baba-66bc4c366885-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0d53207-65f3-4d43-baba-66bc4c366885-tmp-dir\") pod \"b0d53207-65f3-4d43-baba-66bc4c366885\" (UID: \"b0d53207-65f3-4d43-baba-66bc4c366885\") " Oct 30 22:10:20.052052 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:20.052002 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab533734-e0d8-4acb-854d-12851a87bc58-home\" (UniqueName: \"kubernetes.io/empty-dir/ab533734-e0d8-4acb-854d-12851a87bc58-home\") pod \"ab533734-e0d8-4acb-854d-12851a87bc58\" (UID: \"ab533734-e0d8-4acb-854d-12851a87bc58\") " Oct 30 22:10:20.052052 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:20.052001 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0d53207-65f3-4d43-baba-66bc4c366885-model-cache" pod "b0d53207-65f3-4d43-baba-66bc4c366885" (UID: "b0d53207-65f3-4d43-baba-66bc4c366885"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:20.052223 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:20.052060 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0d53207-65f3-4d43-baba-66bc4c366885-tls-certs\" (UniqueName: \"kubernetes.io/secret/b0d53207-65f3-4d43-baba-66bc4c366885-tls-certs\") pod \"b0d53207-65f3-4d43-baba-66bc4c366885\" (UID: \"b0d53207-65f3-4d43-baba-66bc4c366885\") " Oct 30 22:10:20.052223 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:20.052071 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab533734-e0d8-4acb-854d-12851a87bc58-tmp-dir" pod "ab533734-e0d8-4acb-854d-12851a87bc58" (UID: "ab533734-e0d8-4acb-854d-12851a87bc58"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:20.052223 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:20.052088 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab533734-e0d8-4acb-854d-12851a87bc58-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab533734-e0d8-4acb-854d-12851a87bc58-tls-certs\") pod \"ab533734-e0d8-4acb-854d-12851a87bc58\" (UID: \"ab533734-e0d8-4acb-854d-12851a87bc58\") " Oct 30 22:10:20.052223 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:20.052146 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab533734-e0d8-4acb-854d-12851a87bc58-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab533734-e0d8-4acb-854d-12851a87bc58-model-cache\") pod \"ab533734-e0d8-4acb-854d-12851a87bc58\" (UID: \"ab533734-e0d8-4acb-854d-12851a87bc58\") " Oct 30 22:10:20.052223 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:20.052185 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0d53207-65f3-4d43-baba-66bc4c366885-home\" (UniqueName: \"kubernetes.io/empty-dir/b0d53207-65f3-4d43-baba-66bc4c366885-home\") pod \"b0d53207-65f3-4d43-baba-66bc4c366885\" (UID: \"b0d53207-65f3-4d43-baba-66bc4c366885\") " Oct 30 22:10:20.052223 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:20.052210 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0d53207-65f3-4d43-baba-66bc4c366885-kube-api-access-t6s28\" (UniqueName: \"kubernetes.io/projected/b0d53207-65f3-4d43-baba-66bc4c366885-kube-api-access-t6s28\") pod \"b0d53207-65f3-4d43-baba-66bc4c366885\" (UID: \"b0d53207-65f3-4d43-baba-66bc4c366885\") " Oct 30 22:10:20.052540 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:20.052243 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab533734-e0d8-4acb-854d-12851a87bc58-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab533734-e0d8-4acb-854d-12851a87bc58-dshm\") pod \"ab533734-e0d8-4acb-854d-12851a87bc58\" (UID: \"ab533734-e0d8-4acb-854d-12851a87bc58\") " Oct 30 22:10:20.052540 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:20.052283 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab533734-e0d8-4acb-854d-12851a87bc58-home" pod "ab533734-e0d8-4acb-854d-12851a87bc58" (UID: "ab533734-e0d8-4acb-854d-12851a87bc58"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:20.052540 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:20.052498 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab533734-e0d8-4acb-854d-12851a87bc58-model-cache" pod "ab533734-e0d8-4acb-854d-12851a87bc58" (UID: "ab533734-e0d8-4acb-854d-12851a87bc58"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:20.052540 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:20.052514 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0d53207-65f3-4d43-baba-66bc4c366885-tmp-dir" pod "b0d53207-65f3-4d43-baba-66bc4c366885" (UID: "b0d53207-65f3-4d43-baba-66bc4c366885"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:20.052772 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:20.052651 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0d53207-65f3-4d43-baba-66bc4c366885-home" pod "b0d53207-65f3-4d43-baba-66bc4c366885" (UID: "b0d53207-65f3-4d43-baba-66bc4c366885"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:20.053121 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:20.053098 5748 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0d53207-65f3-4d43-baba-66bc4c366885-tmp-dir\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:20.053228 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:20.053125 5748 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab533734-e0d8-4acb-854d-12851a87bc58-home\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:20.053228 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:20.053138 5748 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab533734-e0d8-4acb-854d-12851a87bc58-model-cache\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:20.053228 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:20.053151 5748 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0d53207-65f3-4d43-baba-66bc4c366885-home\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:20.053228 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:20.053164 5748 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0d53207-65f3-4d43-baba-66bc4c366885-model-cache\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:20.053228 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:20.053177 5748 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab533734-e0d8-4acb-854d-12851a87bc58-tmp-dir\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:20.054645 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:20.054619 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0d53207-65f3-4d43-baba-66bc4c366885-dshm" pod "b0d53207-65f3-4d43-baba-66bc4c366885" (UID: "b0d53207-65f3-4d43-baba-66bc4c366885"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:20.054645 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:20.054627 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab533734-e0d8-4acb-854d-12851a87bc58-kube-api-access-8jdf8" pod "ab533734-e0d8-4acb-854d-12851a87bc58" (UID: "ab533734-e0d8-4acb-854d-12851a87bc58"). InnerVolumeSpecName "kube-api-access-8jdf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:20.054785 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:20.054765 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0d53207-65f3-4d43-baba-66bc4c366885-kube-api-access-t6s28" pod "b0d53207-65f3-4d43-baba-66bc4c366885" (UID: "b0d53207-65f3-4d43-baba-66bc4c366885"). InnerVolumeSpecName "kube-api-access-t6s28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:20.055153 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:20.055129 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab533734-e0d8-4acb-854d-12851a87bc58-tls-certs" pod "ab533734-e0d8-4acb-854d-12851a87bc58" (UID: "ab533734-e0d8-4acb-854d-12851a87bc58"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:10:20.055261 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:20.055246 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0d53207-65f3-4d43-baba-66bc4c366885-tls-certs" pod "b0d53207-65f3-4d43-baba-66bc4c366885" (UID: "b0d53207-65f3-4d43-baba-66bc4c366885"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:10:20.055301 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:20.055269 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab533734-e0d8-4acb-854d-12851a87bc58-dshm" pod "ab533734-e0d8-4acb-854d-12851a87bc58" (UID: "ab533734-e0d8-4acb-854d-12851a87bc58"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:20.154681 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:20.154499 5748 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0d53207-65f3-4d43-baba-66bc4c366885-tls-certs\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:20.154681 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:20.154541 5748 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab533734-e0d8-4acb-854d-12851a87bc58-tls-certs\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:20.154681 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:20.154557 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6s28\" (UniqueName: \"kubernetes.io/projected/b0d53207-65f3-4d43-baba-66bc4c366885-kube-api-access-t6s28\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:20.154681 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:20.154573 5748 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab533734-e0d8-4acb-854d-12851a87bc58-dshm\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:20.154681 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:20.154587 5748 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0d53207-65f3-4d43-baba-66bc4c366885-dshm\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:20.154681 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:20.154600 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jdf8\" (UniqueName: \"kubernetes.io/projected/ab533734-e0d8-4acb-854d-12851a87bc58-kube-api-access-8jdf8\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:20.444743 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:20.444660 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-6ea374/simulator-w1-6ea374-kserve-775cfdf98b-p5q25"] Oct 30 22:10:20.448861 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:20.448835 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-6ea374/simulator-w1-6ea374-kserve-775cfdf98b-p5q25"] Oct 30 22:10:20.559826 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:20.559785 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-6ea374/distinct-2-w1-6ea374-kserve-68fb49889-hpg2k"] Oct 30 22:10:20.564029 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:20.563998 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-6ea374/distinct-2-w1-6ea374-kserve-68fb49889-hpg2k"] Oct 30 22:10:21.610725 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:21.610691 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96d4b810-n9xn9"] Oct 30 22:10:21.611136 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:21.611123 5748 cpu_manager.go:414] "containerMap: removing container" podUID="ab533734-e0d8-4acb-854d-12851a87bc58" containerName="main" Oct 30 22:10:21.611178 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:21.611136 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab533734-e0d8-4acb-854d-12851a87bc58" containerName="main" Oct 30 22:10:21.611178 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:21.611160 5748 cpu_manager.go:414] "containerMap: removing container" podUID="b0d53207-65f3-4d43-baba-66bc4c366885" containerName="main" Oct 30 22:10:21.611178 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:21.611166 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0d53207-65f3-4d43-baba-66bc4c366885" containerName="main" Oct 30 22:10:21.611178 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:21.611173 5748 cpu_manager.go:414] "containerMap: removing container" podUID="eed8b35c-e416-4173-98a7-ad7b3d6bc0ea" containerName="payload-processing" Oct 30 22:10:21.611304 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:21.611189 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eed8b35c-e416-4173-98a7-ad7b3d6bc0ea" containerName="payload-processing" Oct 30 22:10:21.611304 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:21.611200 5748 cpu_manager.go:414] "containerMap: removing container" podUID="084e23c1-9a28-4b27-ab32-d3c348fd2c1b" containerName="payload-pre-processing" Oct 30 22:10:21.611304 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:21.611204 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="084e23c1-9a28-4b27-ab32-d3c348fd2c1b" containerName="payload-pre-processing" Oct 30 22:10:21.611304 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:21.611215 5748 cpu_manager.go:414] "containerMap: removing container" podUID="ded0f467-f8cb-4b8e-89e8-fbf29cc3189f" containerName="revoke-keys" Oct 30 22:10:21.611304 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:21.611220 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ded0f467-f8cb-4b8e-89e8-fbf29cc3189f" containerName="revoke-keys" Oct 30 22:10:21.611304 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:21.611291 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="084e23c1-9a28-4b27-ab32-d3c348fd2c1b" containerName="payload-pre-processing" Oct 30 22:10:21.611304 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:21.611299 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="ded0f467-f8cb-4b8e-89e8-fbf29cc3189f" containerName="revoke-keys" Oct 30 22:10:21.611512 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:21.611311 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab533734-e0d8-4acb-854d-12851a87bc58" containerName="main" Oct 30 22:10:21.611512 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:21.611321 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0d53207-65f3-4d43-baba-66bc4c366885" containerName="main" Oct 30 22:10:21.611512 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:21.611326 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="eed8b35c-e416-4173-98a7-ad7b3d6bc0ea" containerName="payload-processing" Oct 30 22:10:21.616025 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:21.616007 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d4b810-n9xn9" Oct 30 22:10:21.670006 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:21.669977 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40bb68cf-74bf-4b69-bc52-7df542f17d48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96d4b810-n9xn9\" (UID: \"40bb68cf-74bf-4b69-bc52-7df542f17d48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d4b810-n9xn9" Oct 30 22:10:21.670150 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:21.670098 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9d9v\" (UniqueName: \"kubernetes.io/projected/40bb68cf-74bf-4b69-bc52-7df542f17d48-kube-api-access-r9d9v\") pod \"maas-api-revoke-sub-96d4b810-n9xn9\" (UID: \"40bb68cf-74bf-4b69-bc52-7df542f17d48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d4b810-n9xn9" Oct 30 22:10:21.670200 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:21.670149 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40bb68cf-74bf-4b69-bc52-7df542f17d48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96d4b810-n9xn9\" (UID: \"40bb68cf-74bf-4b69-bc52-7df542f17d48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d4b810-n9xn9" Oct 30 22:10:21.725048 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:21.725019 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:10:21.725336 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:21.725312 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:10:21.725336 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:21.725328 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:10:21.725523 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:21.725371 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 22:10:21.728177 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:21.728152 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96d4b810-n9xn9"] Oct 30 22:10:21.730663 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:21.730642 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44020668-q6n64"] Oct 30 22:10:21.734432 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:21.734407 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44020668-q6n64" Oct 30 22:10:21.770816 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:21.770792 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40bb68cf-74bf-4b69-bc52-7df542f17d48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96d4b810-n9xn9\" (UID: \"40bb68cf-74bf-4b69-bc52-7df542f17d48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d4b810-n9xn9" Oct 30 22:10:21.770935 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:21.770876 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9d9v\" (UniqueName: \"kubernetes.io/projected/40bb68cf-74bf-4b69-bc52-7df542f17d48-kube-api-access-r9d9v\") pod \"maas-api-revoke-sub-96d4b810-n9xn9\" (UID: \"40bb68cf-74bf-4b69-bc52-7df542f17d48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d4b810-n9xn9" Oct 30 22:10:21.770995 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:21.770936 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40bb68cf-74bf-4b69-bc52-7df542f17d48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96d4b810-n9xn9\" (UID: \"40bb68cf-74bf-4b69-bc52-7df542f17d48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d4b810-n9xn9" Oct 30 22:10:21.771497 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:21.771474 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40bb68cf-74bf-4b69-bc52-7df542f17d48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96d4b810-n9xn9\" (UID: \"40bb68cf-74bf-4b69-bc52-7df542f17d48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d4b810-n9xn9" Oct 30 22:10:21.773072 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:21.773056 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40bb68cf-74bf-4b69-bc52-7df542f17d48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96d4b810-n9xn9\" (UID: \"40bb68cf-74bf-4b69-bc52-7df542f17d48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d4b810-n9xn9" Oct 30 22:10:21.789451 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:21.789431 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44020668-q6n64"] Oct 30 22:10:21.846052 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:21.846030 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9d9v\" (UniqueName: \"kubernetes.io/projected/40bb68cf-74bf-4b69-bc52-7df542f17d48-kube-api-access-r9d9v\") pod \"maas-api-revoke-sub-96d4b810-n9xn9\" (UID: \"40bb68cf-74bf-4b69-bc52-7df542f17d48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d4b810-n9xn9" Oct 30 22:10:21.872266 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:21.872210 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02d5d75d-0e70-43df-9218-ecbb91ea5a59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44020668-q6n64\" (UID: \"02d5d75d-0e70-43df-9218-ecbb91ea5a59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44020668-q6n64" Oct 30 22:10:21.872398 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:21.872377 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mls7h\" (UniqueName: \"kubernetes.io/projected/02d5d75d-0e70-43df-9218-ecbb91ea5a59-kube-api-access-mls7h\") pod \"maas-api-revoke-sub-44020668-q6n64\" (UID: \"02d5d75d-0e70-43df-9218-ecbb91ea5a59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44020668-q6n64" Oct 30 22:10:21.872471 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:21.872430 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02d5d75d-0e70-43df-9218-ecbb91ea5a59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44020668-q6n64\" (UID: \"02d5d75d-0e70-43df-9218-ecbb91ea5a59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44020668-q6n64" Oct 30 22:10:21.929319 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:21.929295 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d4b810-n9xn9" Oct 30 22:10:21.973735 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:21.973698 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mls7h\" (UniqueName: \"kubernetes.io/projected/02d5d75d-0e70-43df-9218-ecbb91ea5a59-kube-api-access-mls7h\") pod \"maas-api-revoke-sub-44020668-q6n64\" (UID: \"02d5d75d-0e70-43df-9218-ecbb91ea5a59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44020668-q6n64" Oct 30 22:10:21.973880 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:21.973750 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02d5d75d-0e70-43df-9218-ecbb91ea5a59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44020668-q6n64\" (UID: \"02d5d75d-0e70-43df-9218-ecbb91ea5a59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44020668-q6n64" Oct 30 22:10:21.973880 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:21.973819 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02d5d75d-0e70-43df-9218-ecbb91ea5a59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44020668-q6n64\" (UID: \"02d5d75d-0e70-43df-9218-ecbb91ea5a59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44020668-q6n64" Oct 30 22:10:21.974399 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:21.974375 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02d5d75d-0e70-43df-9218-ecbb91ea5a59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44020668-q6n64\" (UID: \"02d5d75d-0e70-43df-9218-ecbb91ea5a59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44020668-q6n64" Oct 30 22:10:21.976177 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:21.976148 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02d5d75d-0e70-43df-9218-ecbb91ea5a59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44020668-q6n64\" (UID: \"02d5d75d-0e70-43df-9218-ecbb91ea5a59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44020668-q6n64" Oct 30 22:10:22.037097 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:22.037073 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mls7h\" (UniqueName: \"kubernetes.io/projected/02d5d75d-0e70-43df-9218-ecbb91ea5a59-kube-api-access-mls7h\") pod \"maas-api-revoke-sub-44020668-q6n64\" (UID: \"02d5d75d-0e70-43df-9218-ecbb91ea5a59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44020668-q6n64" Oct 30 22:10:22.047131 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:22.047105 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44020668-q6n64" Oct 30 22:10:22.305752 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:22.305707 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96d4b810-n9xn9"] Oct 30 22:10:22.309411 ip-10-0-143-61 kubenswrapper[5748]: W1030 22:10:22.309381 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40bb68cf_74bf_4b69_bc52_7df542f17d48.slice/crio-91237cf685828ae0adfdf3ab8ecc504b04cb1f80a70f49d320af06321a3f0485 WatchSource:0}: Error finding container 91237cf685828ae0adfdf3ab8ecc504b04cb1f80a70f49d320af06321a3f0485: Status 404 returned error can't find the container with id 91237cf685828ae0adfdf3ab8ecc504b04cb1f80a70f49d320af06321a3f0485 Oct 30 22:10:22.416359 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:22.416333 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab533734-e0d8-4acb-854d-12851a87bc58" path="/var/lib/kubelet/pods/ab533734-e0d8-4acb-854d-12851a87bc58/volumes" Oct 30 22:10:22.416727 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:22.416714 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0d53207-65f3-4d43-baba-66bc4c366885" path="/var/lib/kubelet/pods/b0d53207-65f3-4d43-baba-66bc4c366885/volumes" Oct 30 22:10:22.427391 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:22.427369 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44020668-q6n64"] Oct 30 22:10:22.429251 ip-10-0-143-61 kubenswrapper[5748]: W1030 22:10:22.429226 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02d5d75d_0e70_43df_9218_ecbb91ea5a59.slice/crio-c1030804cd6183054f7169272517c812c07525bff736e6e95430480b229b3eea WatchSource:0}: Error finding container c1030804cd6183054f7169272517c812c07525bff736e6e95430480b229b3eea: Status 404 returned error can't find the container with id c1030804cd6183054f7169272517c812c07525bff736e6e95430480b229b3eea Oct 30 22:10:23.049958 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:23.049913 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d4b810-n9xn9" event={"ID":"40bb68cf-74bf-4b69-bc52-7df542f17d48","Type":"ContainerStarted","Data":"209c37444bc4a54e54664e8607ac5e0303ea084baed33f0fdfe20787587b1351"} Oct 30 22:10:23.050592 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:23.050015 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d4b810-n9xn9" event={"ID":"40bb68cf-74bf-4b69-bc52-7df542f17d48","Type":"ContainerStarted","Data":"91237cf685828ae0adfdf3ab8ecc504b04cb1f80a70f49d320af06321a3f0485"} Oct 30 22:10:23.054699 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:23.054668 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44020668-q6n64" event={"ID":"02d5d75d-0e70-43df-9218-ecbb91ea5a59","Type":"ContainerStarted","Data":"480bff8d12d6fcff995cebae2113f89a9561da1c4fad4826e2ae6343cfa79fae"} Oct 30 22:10:23.054805 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:23.054739 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44020668-q6n64" event={"ID":"02d5d75d-0e70-43df-9218-ecbb91ea5a59","Type":"ContainerStarted","Data":"c1030804cd6183054f7169272517c812c07525bff736e6e95430480b229b3eea"} Oct 30 22:10:23.170455 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:23.170396 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d4b810-n9xn9" podStartSLOduration=2.170381994 podStartE2EDuration="2.170381994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:23.169458209 +0000 UTC m=+2806.911979147" watchObservedRunningTime="2026-10-30 22:10:23.170381994 +0000 UTC m=+2806.912902929" Oct 30 22:10:24.063233 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:24.063197 5748 generic.go:356] "Generic (PLEG): container finished" podID="40bb68cf-74bf-4b69-bc52-7df542f17d48" containerID="209c37444bc4a54e54664e8607ac5e0303ea084baed33f0fdfe20787587b1351" exitCode=0 Oct 30 22:10:24.063673 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:24.063283 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d4b810-n9xn9" event={"ID":"40bb68cf-74bf-4b69-bc52-7df542f17d48","Type":"ContainerDied","Data":"209c37444bc4a54e54664e8607ac5e0303ea084baed33f0fdfe20787587b1351"} Oct 30 22:10:24.065078 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:24.065049 5748 generic.go:356] "Generic (PLEG): container finished" podID="02d5d75d-0e70-43df-9218-ecbb91ea5a59" containerID="480bff8d12d6fcff995cebae2113f89a9561da1c4fad4826e2ae6343cfa79fae" exitCode=0 Oct 30 22:10:24.065225 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:24.065117 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44020668-q6n64" event={"ID":"02d5d75d-0e70-43df-9218-ecbb91ea5a59","Type":"ContainerDied","Data":"480bff8d12d6fcff995cebae2113f89a9561da1c4fad4826e2ae6343cfa79fae"} Oct 30 22:10:24.183280 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:24.183225 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44020668-q6n64" podStartSLOduration=3.183211694 podStartE2EDuration="3.183211694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:23.293048695 +0000 UTC m=+2807.035569630" watchObservedRunningTime="2026-10-30 22:10:24.183211694 +0000 UTC m=+2807.925732629" Oct 30 22:10:25.258480 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:25.258451 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44020668-q6n64" Oct 30 22:10:25.274507 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:25.274475 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d4b810-n9xn9" Oct 30 22:10:25.416383 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:25.416353 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40bb68cf-74bf-4b69-bc52-7df542f17d48-kube-api-access-r9d9v\" (UniqueName: \"kubernetes.io/projected/40bb68cf-74bf-4b69-bc52-7df542f17d48-kube-api-access-r9d9v\") pod \"40bb68cf-74bf-4b69-bc52-7df542f17d48\" (UID: \"40bb68cf-74bf-4b69-bc52-7df542f17d48\") " Oct 30 22:10:25.416579 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:25.416416 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40bb68cf-74bf-4b69-bc52-7df542f17d48-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40bb68cf-74bf-4b69-bc52-7df542f17d48-maas-api-openshift-ca\") pod \"40bb68cf-74bf-4b69-bc52-7df542f17d48\" (UID: \"40bb68cf-74bf-4b69-bc52-7df542f17d48\") " Oct 30 22:10:25.416579 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:25.416449 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02d5d75d-0e70-43df-9218-ecbb91ea5a59-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02d5d75d-0e70-43df-9218-ecbb91ea5a59-maas-api-xks-ca\") pod \"02d5d75d-0e70-43df-9218-ecbb91ea5a59\" (UID: \"02d5d75d-0e70-43df-9218-ecbb91ea5a59\") " Oct 30 22:10:25.416579 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:25.416482 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40bb68cf-74bf-4b69-bc52-7df542f17d48-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40bb68cf-74bf-4b69-bc52-7df542f17d48-maas-api-xks-ca\") pod \"40bb68cf-74bf-4b69-bc52-7df542f17d48\" (UID: \"40bb68cf-74bf-4b69-bc52-7df542f17d48\") " Oct 30 22:10:25.416579 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:25.416560 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02d5d75d-0e70-43df-9218-ecbb91ea5a59-kube-api-access-mls7h\" (UniqueName: \"kubernetes.io/projected/02d5d75d-0e70-43df-9218-ecbb91ea5a59-kube-api-access-mls7h\") pod \"02d5d75d-0e70-43df-9218-ecbb91ea5a59\" (UID: \"02d5d75d-0e70-43df-9218-ecbb91ea5a59\") " Oct 30 22:10:25.416835 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:25.416590 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02d5d75d-0e70-43df-9218-ecbb91ea5a59-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02d5d75d-0e70-43df-9218-ecbb91ea5a59-maas-api-openshift-ca\") pod \"02d5d75d-0e70-43df-9218-ecbb91ea5a59\" (UID: \"02d5d75d-0e70-43df-9218-ecbb91ea5a59\") " Oct 30 22:10:25.416911 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:25.416842 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40bb68cf-74bf-4b69-bc52-7df542f17d48-maas-api-openshift-ca" pod "40bb68cf-74bf-4b69-bc52-7df542f17d48" (UID: "40bb68cf-74bf-4b69-bc52-7df542f17d48"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:10:25.417116 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:25.417084 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40bb68cf-74bf-4b69-bc52-7df542f17d48-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:25.417238 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:25.417200 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02d5d75d-0e70-43df-9218-ecbb91ea5a59-maas-api-openshift-ca" pod "02d5d75d-0e70-43df-9218-ecbb91ea5a59" (UID: "02d5d75d-0e70-43df-9218-ecbb91ea5a59"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:10:25.419163 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:25.419138 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40bb68cf-74bf-4b69-bc52-7df542f17d48-maas-api-xks-ca" pod "40bb68cf-74bf-4b69-bc52-7df542f17d48" (UID: "40bb68cf-74bf-4b69-bc52-7df542f17d48"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:10:25.419265 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:25.419217 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02d5d75d-0e70-43df-9218-ecbb91ea5a59-maas-api-xks-ca" pod "02d5d75d-0e70-43df-9218-ecbb91ea5a59" (UID: "02d5d75d-0e70-43df-9218-ecbb91ea5a59"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:10:25.419351 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:25.419329 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02d5d75d-0e70-43df-9218-ecbb91ea5a59-kube-api-access-mls7h" pod "02d5d75d-0e70-43df-9218-ecbb91ea5a59" (UID: "02d5d75d-0e70-43df-9218-ecbb91ea5a59"). InnerVolumeSpecName "kube-api-access-mls7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:25.419409 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:25.419350 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40bb68cf-74bf-4b69-bc52-7df542f17d48-kube-api-access-r9d9v" pod "40bb68cf-74bf-4b69-bc52-7df542f17d48" (UID: "40bb68cf-74bf-4b69-bc52-7df542f17d48"). InnerVolumeSpecName "kube-api-access-r9d9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:25.518404 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:25.518367 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02d5d75d-0e70-43df-9218-ecbb91ea5a59-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:25.518568 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:25.518409 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40bb68cf-74bf-4b69-bc52-7df542f17d48-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:25.518568 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:25.518435 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mls7h\" (UniqueName: \"kubernetes.io/projected/02d5d75d-0e70-43df-9218-ecbb91ea5a59-kube-api-access-mls7h\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:25.518568 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:25.518450 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02d5d75d-0e70-43df-9218-ecbb91ea5a59-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:25.518568 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:25.518464 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9d9v\" (UniqueName: \"kubernetes.io/projected/40bb68cf-74bf-4b69-bc52-7df542f17d48-kube-api-access-r9d9v\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:26.084197 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:26.084158 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d4b810-n9xn9" event={"ID":"40bb68cf-74bf-4b69-bc52-7df542f17d48","Type":"ContainerDied","Data":"91237cf685828ae0adfdf3ab8ecc504b04cb1f80a70f49d320af06321a3f0485"} Oct 30 22:10:26.084458 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:26.084222 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91237cf685828ae0adfdf3ab8ecc504b04cb1f80a70f49d320af06321a3f0485" Oct 30 22:10:26.084458 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:26.084166 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d4b810-n9xn9" Oct 30 22:10:26.085632 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:26.085611 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44020668-q6n64" Oct 30 22:10:26.085779 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:26.085640 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44020668-q6n64" event={"ID":"02d5d75d-0e70-43df-9218-ecbb91ea5a59","Type":"ContainerDied","Data":"c1030804cd6183054f7169272517c812c07525bff736e6e95430480b229b3eea"} Oct 30 22:10:26.085779 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:26.085688 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1030804cd6183054f7169272517c812c07525bff736e6e95430480b229b3eea" Oct 30 22:10:27.161502 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:27.161467 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2566e01-flg2f"] Oct 30 22:10:27.162126 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:27.162103 5748 cpu_manager.go:414] "containerMap: removing container" podUID="02d5d75d-0e70-43df-9218-ecbb91ea5a59" containerName="revoke-keys" Oct 30 22:10:27.162126 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:27.162119 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02d5d75d-0e70-43df-9218-ecbb91ea5a59" containerName="revoke-keys" Oct 30 22:10:27.162239 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:27.162151 5748 cpu_manager.go:414] "containerMap: removing container" podUID="40bb68cf-74bf-4b69-bc52-7df542f17d48" containerName="revoke-keys" Oct 30 22:10:27.162239 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:27.162157 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40bb68cf-74bf-4b69-bc52-7df542f17d48" containerName="revoke-keys" Oct 30 22:10:27.162318 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:27.162248 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="40bb68cf-74bf-4b69-bc52-7df542f17d48" containerName="revoke-keys" Oct 30 22:10:27.162318 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:27.162264 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="02d5d75d-0e70-43df-9218-ecbb91ea5a59" containerName="revoke-keys" Oct 30 22:10:27.165879 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:27.165859 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2566e01-flg2f" Oct 30 22:10:27.227888 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:27.227844 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4aa6bbe1-zgmlk"] Oct 30 22:10:27.232420 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:27.232390 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2566e01-flg2f"] Oct 30 22:10:27.232572 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:27.232516 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aa6bbe1-zgmlk" Oct 30 22:10:27.269093 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:27.269059 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:10:27.269093 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:27.269074 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 22:10:27.269093 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:27.269079 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:10:27.269300 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:27.269067 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:10:27.280752 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:27.280728 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4aa6bbe1-zgmlk"] Oct 30 22:10:27.336603 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:27.336573 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bfbddd0-67de-45c5-bb50-7f809c330b3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2566e01-flg2f\" (UID: \"5bfbddd0-67de-45c5-bb50-7f809c330b3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2566e01-flg2f" Oct 30 22:10:27.336777 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:27.336614 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/123c12b7-4ff3-435a-8a6d-593218afbd35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4aa6bbe1-zgmlk\" (UID: \"123c12b7-4ff3-435a-8a6d-593218afbd35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aa6bbe1-zgmlk" Oct 30 22:10:27.336777 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:27.336641 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdjjv\" (UniqueName: \"kubernetes.io/projected/123c12b7-4ff3-435a-8a6d-593218afbd35-kube-api-access-xdjjv\") pod \"maas-api-revoke-sub-4aa6bbe1-zgmlk\" (UID: \"123c12b7-4ff3-435a-8a6d-593218afbd35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aa6bbe1-zgmlk" Oct 30 22:10:27.336859 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:27.336776 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bfbddd0-67de-45c5-bb50-7f809c330b3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2566e01-flg2f\" (UID: \"5bfbddd0-67de-45c5-bb50-7f809c330b3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2566e01-flg2f" Oct 30 22:10:27.336859 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:27.336842 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/123c12b7-4ff3-435a-8a6d-593218afbd35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4aa6bbe1-zgmlk\" (UID: \"123c12b7-4ff3-435a-8a6d-593218afbd35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aa6bbe1-zgmlk" Oct 30 22:10:27.336972 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:27.336960 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tj8n\" (UniqueName: \"kubernetes.io/projected/5bfbddd0-67de-45c5-bb50-7f809c330b3e-kube-api-access-4tj8n\") pod \"maas-api-revoke-sub-e2566e01-flg2f\" (UID: \"5bfbddd0-67de-45c5-bb50-7f809c330b3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2566e01-flg2f" Oct 30 22:10:27.438279 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:27.438200 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bfbddd0-67de-45c5-bb50-7f809c330b3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2566e01-flg2f\" (UID: \"5bfbddd0-67de-45c5-bb50-7f809c330b3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2566e01-flg2f" Oct 30 22:10:27.438279 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:27.438241 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/123c12b7-4ff3-435a-8a6d-593218afbd35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4aa6bbe1-zgmlk\" (UID: \"123c12b7-4ff3-435a-8a6d-593218afbd35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aa6bbe1-zgmlk" Oct 30 22:10:27.438279 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:27.438267 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdjjv\" (UniqueName: \"kubernetes.io/projected/123c12b7-4ff3-435a-8a6d-593218afbd35-kube-api-access-xdjjv\") pod \"maas-api-revoke-sub-4aa6bbe1-zgmlk\" (UID: \"123c12b7-4ff3-435a-8a6d-593218afbd35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aa6bbe1-zgmlk" Oct 30 22:10:27.438591 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:27.438309 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bfbddd0-67de-45c5-bb50-7f809c330b3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2566e01-flg2f\" (UID: \"5bfbddd0-67de-45c5-bb50-7f809c330b3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2566e01-flg2f" Oct 30 22:10:27.438591 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:27.438348 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/123c12b7-4ff3-435a-8a6d-593218afbd35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4aa6bbe1-zgmlk\" (UID: \"123c12b7-4ff3-435a-8a6d-593218afbd35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aa6bbe1-zgmlk" Oct 30 22:10:27.438591 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:27.438407 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tj8n\" (UniqueName: \"kubernetes.io/projected/5bfbddd0-67de-45c5-bb50-7f809c330b3e-kube-api-access-4tj8n\") pod \"maas-api-revoke-sub-e2566e01-flg2f\" (UID: \"5bfbddd0-67de-45c5-bb50-7f809c330b3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2566e01-flg2f" Oct 30 22:10:27.439017 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:27.438988 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/123c12b7-4ff3-435a-8a6d-593218afbd35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4aa6bbe1-zgmlk\" (UID: \"123c12b7-4ff3-435a-8a6d-593218afbd35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aa6bbe1-zgmlk" Oct 30 22:10:27.439154 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:27.439030 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bfbddd0-67de-45c5-bb50-7f809c330b3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2566e01-flg2f\" (UID: \"5bfbddd0-67de-45c5-bb50-7f809c330b3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2566e01-flg2f" Oct 30 22:10:27.440680 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:27.440649 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bfbddd0-67de-45c5-bb50-7f809c330b3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2566e01-flg2f\" (UID: \"5bfbddd0-67de-45c5-bb50-7f809c330b3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2566e01-flg2f" Oct 30 22:10:27.440774 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:27.440660 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/123c12b7-4ff3-435a-8a6d-593218afbd35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4aa6bbe1-zgmlk\" (UID: \"123c12b7-4ff3-435a-8a6d-593218afbd35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aa6bbe1-zgmlk" Oct 30 22:10:27.508175 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:27.508140 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdjjv\" (UniqueName: \"kubernetes.io/projected/123c12b7-4ff3-435a-8a6d-593218afbd35-kube-api-access-xdjjv\") pod \"maas-api-revoke-sub-4aa6bbe1-zgmlk\" (UID: \"123c12b7-4ff3-435a-8a6d-593218afbd35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aa6bbe1-zgmlk" Oct 30 22:10:27.508330 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:27.508182 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tj8n\" (UniqueName: \"kubernetes.io/projected/5bfbddd0-67de-45c5-bb50-7f809c330b3e-kube-api-access-4tj8n\") pod \"maas-api-revoke-sub-e2566e01-flg2f\" (UID: \"5bfbddd0-67de-45c5-bb50-7f809c330b3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2566e01-flg2f" Oct 30 22:10:27.546890 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:27.546858 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aa6bbe1-zgmlk" Oct 30 22:10:27.779544 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:27.779465 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2566e01-flg2f" Oct 30 22:10:27.926962 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:27.926936 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4aa6bbe1-zgmlk"] Oct 30 22:10:27.929011 ip-10-0-143-61 kubenswrapper[5748]: W1030 22:10:27.928983 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod123c12b7_4ff3_435a_8a6d_593218afbd35.slice/crio-772029fe7db6b0907642b0c563f6fd72511fad1988883b964786f01cda5dea44 WatchSource:0}: Error finding container 772029fe7db6b0907642b0c563f6fd72511fad1988883b964786f01cda5dea44: Status 404 returned error can't find the container with id 772029fe7db6b0907642b0c563f6fd72511fad1988883b964786f01cda5dea44 Oct 30 22:10:28.111086 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:28.111018 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aa6bbe1-zgmlk" event={"ID":"123c12b7-4ff3-435a-8a6d-593218afbd35","Type":"ContainerStarted","Data":"772029fe7db6b0907642b0c563f6fd72511fad1988883b964786f01cda5dea44"} Oct 30 22:10:28.163619 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:28.163576 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2566e01-flg2f"] Oct 30 22:10:29.119209 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:29.119177 5748 generic.go:356] "Generic (PLEG): container finished" podID="123c12b7-4ff3-435a-8a6d-593218afbd35" containerID="5f31883435d611abedfa0b34d92224cb8c31d0f4d35c2d431cc5a293face6664" exitCode=0 Oct 30 22:10:29.119399 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:29.119259 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aa6bbe1-zgmlk" event={"ID":"123c12b7-4ff3-435a-8a6d-593218afbd35","Type":"ContainerDied","Data":"5f31883435d611abedfa0b34d92224cb8c31d0f4d35c2d431cc5a293face6664"} Oct 30 22:10:29.120947 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:29.120922 5748 generic.go:356] "Generic (PLEG): container finished" podID="5bfbddd0-67de-45c5-bb50-7f809c330b3e" containerID="564f8c162f64c5d4a00c67df0bbfc9cd856ceac08984c3b95facf4cd3536d865" exitCode=0 Oct 30 22:10:29.120947 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:29.120923 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2566e01-flg2f" event={"ID":"5bfbddd0-67de-45c5-bb50-7f809c330b3e","Type":"ContainerDied","Data":"564f8c162f64c5d4a00c67df0bbfc9cd856ceac08984c3b95facf4cd3536d865"} Oct 30 22:10:29.121108 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:29.120979 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2566e01-flg2f" event={"ID":"5bfbddd0-67de-45c5-bb50-7f809c330b3e","Type":"ContainerStarted","Data":"d9c0d10f22b0892d6be42f2acee1e584ad2874bae424dc376ba0b4d9f324aee9"} Oct 30 22:10:30.149235 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:10:30.149159 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:10:30.292827 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:30.292805 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aa6bbe1-zgmlk" Oct 30 22:10:30.295926 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:30.295885 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2566e01-flg2f" Oct 30 22:10:30.469652 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:30.469623 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/123c12b7-4ff3-435a-8a6d-593218afbd35-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/123c12b7-4ff3-435a-8a6d-593218afbd35-maas-api-xks-ca\") pod \"123c12b7-4ff3-435a-8a6d-593218afbd35\" (UID: \"123c12b7-4ff3-435a-8a6d-593218afbd35\") " Oct 30 22:10:30.469842 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:30.469730 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5bfbddd0-67de-45c5-bb50-7f809c330b3e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bfbddd0-67de-45c5-bb50-7f809c330b3e-maas-api-xks-ca\") pod \"5bfbddd0-67de-45c5-bb50-7f809c330b3e\" (UID: \"5bfbddd0-67de-45c5-bb50-7f809c330b3e\") " Oct 30 22:10:30.469842 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:30.469761 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/123c12b7-4ff3-435a-8a6d-593218afbd35-kube-api-access-xdjjv\" (UniqueName: \"kubernetes.io/projected/123c12b7-4ff3-435a-8a6d-593218afbd35-kube-api-access-xdjjv\") pod \"123c12b7-4ff3-435a-8a6d-593218afbd35\" (UID: \"123c12b7-4ff3-435a-8a6d-593218afbd35\") " Oct 30 22:10:30.469842 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:30.469796 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bfbddd0-67de-45c5-bb50-7f809c330b3e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bfbddd0-67de-45c5-bb50-7f809c330b3e-maas-api-openshift-ca\") pod \"5bfbddd0-67de-45c5-bb50-7f809c330b3e\" (UID: \"5bfbddd0-67de-45c5-bb50-7f809c330b3e\") " Oct 30 22:10:30.469842 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:30.469823 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/123c12b7-4ff3-435a-8a6d-593218afbd35-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/123c12b7-4ff3-435a-8a6d-593218afbd35-maas-api-openshift-ca\") pod \"123c12b7-4ff3-435a-8a6d-593218afbd35\" (UID: \"123c12b7-4ff3-435a-8a6d-593218afbd35\") " Oct 30 22:10:30.470139 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:30.469932 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bfbddd0-67de-45c5-bb50-7f809c330b3e-kube-api-access-4tj8n\" (UniqueName: \"kubernetes.io/projected/5bfbddd0-67de-45c5-bb50-7f809c330b3e-kube-api-access-4tj8n\") pod \"5bfbddd0-67de-45c5-bb50-7f809c330b3e\" (UID: \"5bfbddd0-67de-45c5-bb50-7f809c330b3e\") " Oct 30 22:10:30.470218 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:30.470181 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bfbddd0-67de-45c5-bb50-7f809c330b3e-maas-api-openshift-ca" pod "5bfbddd0-67de-45c5-bb50-7f809c330b3e" (UID: "5bfbddd0-67de-45c5-bb50-7f809c330b3e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:10:30.470364 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:30.470317 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bfbddd0-67de-45c5-bb50-7f809c330b3e-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:30.470364 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:30.470303 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/123c12b7-4ff3-435a-8a6d-593218afbd35-maas-api-openshift-ca" pod "123c12b7-4ff3-435a-8a6d-593218afbd35" (UID: "123c12b7-4ff3-435a-8a6d-593218afbd35"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:10:30.471739 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:30.471713 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/123c12b7-4ff3-435a-8a6d-593218afbd35-maas-api-xks-ca" pod "123c12b7-4ff3-435a-8a6d-593218afbd35" (UID: "123c12b7-4ff3-435a-8a6d-593218afbd35"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:10:30.471859 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:30.471831 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bfbddd0-67de-45c5-bb50-7f809c330b3e-maas-api-xks-ca" pod "5bfbddd0-67de-45c5-bb50-7f809c330b3e" (UID: "5bfbddd0-67de-45c5-bb50-7f809c330b3e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:10:30.472261 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:30.472241 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/123c12b7-4ff3-435a-8a6d-593218afbd35-kube-api-access-xdjjv" pod "123c12b7-4ff3-435a-8a6d-593218afbd35" (UID: "123c12b7-4ff3-435a-8a6d-593218afbd35"). InnerVolumeSpecName "kube-api-access-xdjjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:30.472346 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:30.472257 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bfbddd0-67de-45c5-bb50-7f809c330b3e-kube-api-access-4tj8n" pod "5bfbddd0-67de-45c5-bb50-7f809c330b3e" (UID: "5bfbddd0-67de-45c5-bb50-7f809c330b3e"). InnerVolumeSpecName "kube-api-access-4tj8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:30.517658 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:30.517609 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-ea1f0b/test-model-ea1f0b-kserve-586698c74f-dmv5g" podUID="a8257bca-a09a-415f-9616-86bdc26b79b1" containerName="main" containerID="cri-o://14d286bb030531ec871ffbbb5dd48aa91b1fb5fdad5ffb8578762783cd0e5e34" gracePeriod=45 Oct 30 22:10:30.571180 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:30.571148 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tj8n\" (UniqueName: \"kubernetes.io/projected/5bfbddd0-67de-45c5-bb50-7f809c330b3e-kube-api-access-4tj8n\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:30.571180 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:30.571177 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/123c12b7-4ff3-435a-8a6d-593218afbd35-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:30.571409 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:30.571187 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bfbddd0-67de-45c5-bb50-7f809c330b3e-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:30.571409 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:30.571199 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdjjv\" (UniqueName: \"kubernetes.io/projected/123c12b7-4ff3-435a-8a6d-593218afbd35-kube-api-access-xdjjv\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:30.571409 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:30.571208 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/123c12b7-4ff3-435a-8a6d-593218afbd35-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:30.757263 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:30.757237 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ea1f0b/test-model-ea1f0b-kserve-586698c74f-dmv5g" Oct 30 22:10:30.874164 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:30.874129 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8257bca-a09a-415f-9616-86bdc26b79b1-kube-api-access-gpkt8\" (UniqueName: \"kubernetes.io/projected/a8257bca-a09a-415f-9616-86bdc26b79b1-kube-api-access-gpkt8\") pod \"a8257bca-a09a-415f-9616-86bdc26b79b1\" (UID: \"a8257bca-a09a-415f-9616-86bdc26b79b1\") " Oct 30 22:10:30.874347 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:30.874186 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8257bca-a09a-415f-9616-86bdc26b79b1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8257bca-a09a-415f-9616-86bdc26b79b1-dshm\") pod \"a8257bca-a09a-415f-9616-86bdc26b79b1\" (UID: \"a8257bca-a09a-415f-9616-86bdc26b79b1\") " Oct 30 22:10:30.874347 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:30.874282 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8257bca-a09a-415f-9616-86bdc26b79b1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8257bca-a09a-415f-9616-86bdc26b79b1-model-cache\") pod \"a8257bca-a09a-415f-9616-86bdc26b79b1\" (UID: \"a8257bca-a09a-415f-9616-86bdc26b79b1\") " Oct 30 22:10:30.874486 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:30.874364 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8257bca-a09a-415f-9616-86bdc26b79b1-home\" (UniqueName: \"kubernetes.io/empty-dir/a8257bca-a09a-415f-9616-86bdc26b79b1-home\") pod \"a8257bca-a09a-415f-9616-86bdc26b79b1\" (UID: \"a8257bca-a09a-415f-9616-86bdc26b79b1\") " Oct 30 22:10:30.874486 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:30.874407 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8257bca-a09a-415f-9616-86bdc26b79b1-tls-certs\" (UniqueName: \"kubernetes.io/secret/a8257bca-a09a-415f-9616-86bdc26b79b1-tls-certs\") pod \"a8257bca-a09a-415f-9616-86bdc26b79b1\" (UID: \"a8257bca-a09a-415f-9616-86bdc26b79b1\") " Oct 30 22:10:30.874486 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:30.874454 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8257bca-a09a-415f-9616-86bdc26b79b1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8257bca-a09a-415f-9616-86bdc26b79b1-tmp-dir\") pod \"a8257bca-a09a-415f-9616-86bdc26b79b1\" (UID: \"a8257bca-a09a-415f-9616-86bdc26b79b1\") " Oct 30 22:10:30.874662 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:30.874482 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8257bca-a09a-415f-9616-86bdc26b79b1-model-cache" pod "a8257bca-a09a-415f-9616-86bdc26b79b1" (UID: "a8257bca-a09a-415f-9616-86bdc26b79b1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:30.874662 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:30.874643 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8257bca-a09a-415f-9616-86bdc26b79b1-home" pod "a8257bca-a09a-415f-9616-86bdc26b79b1" (UID: "a8257bca-a09a-415f-9616-86bdc26b79b1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:30.874761 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:30.874741 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8257bca-a09a-415f-9616-86bdc26b79b1-tmp-dir" pod "a8257bca-a09a-415f-9616-86bdc26b79b1" (UID: "a8257bca-a09a-415f-9616-86bdc26b79b1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:30.874965 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:30.874945 5748 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8257bca-a09a-415f-9616-86bdc26b79b1-tmp-dir\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:30.875075 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:30.874968 5748 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8257bca-a09a-415f-9616-86bdc26b79b1-model-cache\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:30.875075 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:30.874981 5748 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8257bca-a09a-415f-9616-86bdc26b79b1-home\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:30.877137 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:30.877115 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8257bca-a09a-415f-9616-86bdc26b79b1-kube-api-access-gpkt8" pod "a8257bca-a09a-415f-9616-86bdc26b79b1" (UID: "a8257bca-a09a-415f-9616-86bdc26b79b1"). InnerVolumeSpecName "kube-api-access-gpkt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:30.877236 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:30.877162 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8257bca-a09a-415f-9616-86bdc26b79b1-tls-certs" pod "a8257bca-a09a-415f-9616-86bdc26b79b1" (UID: "a8257bca-a09a-415f-9616-86bdc26b79b1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:10:30.877236 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:30.877172 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8257bca-a09a-415f-9616-86bdc26b79b1-dshm" pod "a8257bca-a09a-415f-9616-86bdc26b79b1" (UID: "a8257bca-a09a-415f-9616-86bdc26b79b1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:30.975963 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:30.975933 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpkt8\" (UniqueName: \"kubernetes.io/projected/a8257bca-a09a-415f-9616-86bdc26b79b1-kube-api-access-gpkt8\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:30.975963 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:30.975957 5748 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8257bca-a09a-415f-9616-86bdc26b79b1-dshm\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:30.975963 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:30.975967 5748 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8257bca-a09a-415f-9616-86bdc26b79b1-tls-certs\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:31.135335 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:31.135251 5748 generic.go:356] "Generic (PLEG): container finished" podID="a8257bca-a09a-415f-9616-86bdc26b79b1" containerID="14d286bb030531ec871ffbbb5dd48aa91b1fb5fdad5ffb8578762783cd0e5e34" exitCode=0 Oct 30 22:10:31.135335 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:31.135292 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ea1f0b/test-model-ea1f0b-kserve-586698c74f-dmv5g" Oct 30 22:10:31.135578 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:31.135339 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ea1f0b/test-model-ea1f0b-kserve-586698c74f-dmv5g" event={"ID":"a8257bca-a09a-415f-9616-86bdc26b79b1","Type":"ContainerDied","Data":"14d286bb030531ec871ffbbb5dd48aa91b1fb5fdad5ffb8578762783cd0e5e34"} Oct 30 22:10:31.135578 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:31.135388 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ea1f0b/test-model-ea1f0b-kserve-586698c74f-dmv5g" event={"ID":"a8257bca-a09a-415f-9616-86bdc26b79b1","Type":"ContainerDied","Data":"ff2aeda6e7a878ec075a00b7e9cc0604365556953991a7006c44335c95a77f46"} Oct 30 22:10:31.135578 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:31.135410 5748 scope.go:122] "RemoveContainer" containerID="14d286bb030531ec871ffbbb5dd48aa91b1fb5fdad5ffb8578762783cd0e5e34" Oct 30 22:10:31.136826 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:31.136776 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2566e01-flg2f" Oct 30 22:10:31.136826 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:31.136809 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2566e01-flg2f" event={"ID":"5bfbddd0-67de-45c5-bb50-7f809c330b3e","Type":"ContainerDied","Data":"d9c0d10f22b0892d6be42f2acee1e584ad2874bae424dc376ba0b4d9f324aee9"} Oct 30 22:10:31.137014 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:31.136854 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9c0d10f22b0892d6be42f2acee1e584ad2874bae424dc376ba0b4d9f324aee9" Oct 30 22:10:31.138375 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:31.138355 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aa6bbe1-zgmlk" Oct 30 22:10:31.138513 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:31.138354 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aa6bbe1-zgmlk" event={"ID":"123c12b7-4ff3-435a-8a6d-593218afbd35","Type":"ContainerDied","Data":"772029fe7db6b0907642b0c563f6fd72511fad1988883b964786f01cda5dea44"} Oct 30 22:10:31.138513 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:31.138505 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="772029fe7db6b0907642b0c563f6fd72511fad1988883b964786f01cda5dea44" Oct 30 22:10:31.145947 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:31.145931 5748 scope.go:122] "RemoveContainer" containerID="14d286bb030531ec871ffbbb5dd48aa91b1fb5fdad5ffb8578762783cd0e5e34" Oct 30 22:10:31.146251 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:10:31.146220 5748 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14d286bb030531ec871ffbbb5dd48aa91b1fb5fdad5ffb8578762783cd0e5e34\": container with ID starting with 14d286bb030531ec871ffbbb5dd48aa91b1fb5fdad5ffb8578762783cd0e5e34 not found: ID does not exist" containerID="14d286bb030531ec871ffbbb5dd48aa91b1fb5fdad5ffb8578762783cd0e5e34" Oct 30 22:10:31.146377 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:31.146260 5748 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14d286bb030531ec871ffbbb5dd48aa91b1fb5fdad5ffb8578762783cd0e5e34"} err="failed to get container status \"14d286bb030531ec871ffbbb5dd48aa91b1fb5fdad5ffb8578762783cd0e5e34\": rpc error: code = NotFound desc = could not find container \"14d286bb030531ec871ffbbb5dd48aa91b1fb5fdad5ffb8578762783cd0e5e34\": container with ID starting with 14d286bb030531ec871ffbbb5dd48aa91b1fb5fdad5ffb8578762783cd0e5e34 not found: ID does not exist" Oct 30 22:10:31.357042 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:31.357007 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ea1f0b/test-model-ea1f0b-kserve-586698c74f-dmv5g"] Oct 30 22:10:31.407507 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:31.407475 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-ea1f0b/test-model-ea1f0b-kserve-586698c74f-dmv5g"] Oct 30 22:10:32.416399 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:32.416266 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8257bca-a09a-415f-9616-86bdc26b79b1" path="/var/lib/kubelet/pods/a8257bca-a09a-415f-9616-86bdc26b79b1/volumes" Oct 30 22:10:39.066392 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:39.066341 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-836152a7-f89tv"] Oct 30 22:10:39.116781 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:39.116743 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d979ec1-rkz69"] Oct 30 22:10:39.118941 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:39.118916 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-836152a7-f89tv"] Oct 30 22:10:39.120559 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:39.120538 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d979ec1-rkz69"] Oct 30 22:10:40.418174 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:40.418138 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a05e77b5-b247-40a2-b6e3-9f75b57feacb" path="/var/lib/kubelet/pods/a05e77b5-b247-40a2-b6e3-9f75b57feacb/volumes" Oct 30 22:10:40.419297 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:40.419274 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac81c948-6995-40f2-acbe-ee1919a61e0c" path="/var/lib/kubelet/pods/ac81c948-6995-40f2-acbe-ee1919a61e0c/volumes" Oct 30 22:10:51.874243 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:51.874198 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ce82c78-f9wrj"] Oct 30 22:10:51.874930 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:51.874892 5748 cpu_manager.go:414] "containerMap: removing container" podUID="a8257bca-a09a-415f-9616-86bdc26b79b1" containerName="main" Oct 30 22:10:51.874987 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:51.874934 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8257bca-a09a-415f-9616-86bdc26b79b1" containerName="main" Oct 30 22:10:51.874987 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:51.874968 5748 cpu_manager.go:414] "containerMap: removing container" podUID="123c12b7-4ff3-435a-8a6d-593218afbd35" containerName="revoke-keys" Oct 30 22:10:51.874987 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:51.874977 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="123c12b7-4ff3-435a-8a6d-593218afbd35" containerName="revoke-keys" Oct 30 22:10:51.875088 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:51.874989 5748 cpu_manager.go:414] "containerMap: removing container" podUID="5bfbddd0-67de-45c5-bb50-7f809c330b3e" containerName="revoke-keys" Oct 30 22:10:51.875088 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:51.874999 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bfbddd0-67de-45c5-bb50-7f809c330b3e" containerName="revoke-keys" Oct 30 22:10:51.875163 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:51.875150 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8257bca-a09a-415f-9616-86bdc26b79b1" containerName="main" Oct 30 22:10:51.875198 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:51.875174 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="123c12b7-4ff3-435a-8a6d-593218afbd35" containerName="revoke-keys" Oct 30 22:10:51.875198 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:51.875185 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bfbddd0-67de-45c5-bb50-7f809c330b3e" containerName="revoke-keys" Oct 30 22:10:51.880689 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:51.880668 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce82c78-f9wrj" Oct 30 22:10:51.936312 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:51.936278 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ce82c78-f9wrj"] Oct 30 22:10:51.983535 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:51.983503 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:10:51.983535 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:51.983504 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:10:51.983733 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:51.983571 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 22:10:51.984324 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:51.984303 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:10:51.992484 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:51.992461 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89a64fcf-55f5-4d7f-8a79-f97d61b582d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ce82c78-f9wrj\" (UID: \"89a64fcf-55f5-4d7f-8a79-f97d61b582d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce82c78-f9wrj" Oct 30 22:10:51.992567 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:51.992501 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdls8\" (UniqueName: \"kubernetes.io/projected/89a64fcf-55f5-4d7f-8a79-f97d61b582d1-kube-api-access-mdls8\") pod \"maas-api-revoke-sub-7ce82c78-f9wrj\" (UID: \"89a64fcf-55f5-4d7f-8a79-f97d61b582d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce82c78-f9wrj" Oct 30 22:10:51.992567 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:51.992519 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89a64fcf-55f5-4d7f-8a79-f97d61b582d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ce82c78-f9wrj\" (UID: \"89a64fcf-55f5-4d7f-8a79-f97d61b582d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce82c78-f9wrj" Oct 30 22:10:52.094206 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:52.094176 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89a64fcf-55f5-4d7f-8a79-f97d61b582d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ce82c78-f9wrj\" (UID: \"89a64fcf-55f5-4d7f-8a79-f97d61b582d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce82c78-f9wrj" Oct 30 22:10:52.094399 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:52.094219 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdls8\" (UniqueName: \"kubernetes.io/projected/89a64fcf-55f5-4d7f-8a79-f97d61b582d1-kube-api-access-mdls8\") pod \"maas-api-revoke-sub-7ce82c78-f9wrj\" (UID: \"89a64fcf-55f5-4d7f-8a79-f97d61b582d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce82c78-f9wrj" Oct 30 22:10:52.094399 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:52.094238 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89a64fcf-55f5-4d7f-8a79-f97d61b582d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ce82c78-f9wrj\" (UID: \"89a64fcf-55f5-4d7f-8a79-f97d61b582d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce82c78-f9wrj" Oct 30 22:10:52.094914 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:52.094878 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89a64fcf-55f5-4d7f-8a79-f97d61b582d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ce82c78-f9wrj\" (UID: \"89a64fcf-55f5-4d7f-8a79-f97d61b582d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce82c78-f9wrj" Oct 30 22:10:52.096575 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:52.096549 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89a64fcf-55f5-4d7f-8a79-f97d61b582d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ce82c78-f9wrj\" (UID: \"89a64fcf-55f5-4d7f-8a79-f97d61b582d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce82c78-f9wrj" Oct 30 22:10:52.156436 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:52.156396 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdls8\" (UniqueName: \"kubernetes.io/projected/89a64fcf-55f5-4d7f-8a79-f97d61b582d1-kube-api-access-mdls8\") pod \"maas-api-revoke-sub-7ce82c78-f9wrj\" (UID: \"89a64fcf-55f5-4d7f-8a79-f97d61b582d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce82c78-f9wrj" Oct 30 22:10:52.197305 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:52.197267 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce82c78-f9wrj" Oct 30 22:10:52.591263 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:52.591238 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ce82c78-f9wrj"] Oct 30 22:10:52.594347 ip-10-0-143-61 kubenswrapper[5748]: W1030 22:10:52.594316 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89a64fcf_55f5_4d7f_8a79_f97d61b582d1.slice/crio-03888395dffe87a73c4d1532c36b5f208cd683e0807e0648428fc5f37cbdc8cf WatchSource:0}: Error finding container 03888395dffe87a73c4d1532c36b5f208cd683e0807e0648428fc5f37cbdc8cf: Status 404 returned error can't find the container with id 03888395dffe87a73c4d1532c36b5f208cd683e0807e0648428fc5f37cbdc8cf Oct 30 22:10:53.285392 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:53.285304 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce82c78-f9wrj" event={"ID":"89a64fcf-55f5-4d7f-8a79-f97d61b582d1","Type":"ContainerStarted","Data":"c8ad86110fc66df2ee6f01493035a478ad7d68a82d6910414ca9dd893bbfe5cc"} Oct 30 22:10:53.285392 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:53.285365 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce82c78-f9wrj" event={"ID":"89a64fcf-55f5-4d7f-8a79-f97d61b582d1","Type":"ContainerStarted","Data":"03888395dffe87a73c4d1532c36b5f208cd683e0807e0648428fc5f37cbdc8cf"} Oct 30 22:10:53.404312 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:53.404259 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce82c78-f9wrj" podStartSLOduration=2.404245045 podStartE2EDuration="2.404245045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:53.402389584 +0000 UTC m=+2837.144910540" watchObservedRunningTime="2026-10-30 22:10:53.404245045 +0000 UTC m=+2837.146765979" Oct 30 22:10:54.292331 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:54.292282 5748 generic.go:356] "Generic (PLEG): container finished" podID="89a64fcf-55f5-4d7f-8a79-f97d61b582d1" containerID="c8ad86110fc66df2ee6f01493035a478ad7d68a82d6910414ca9dd893bbfe5cc" exitCode=0 Oct 30 22:10:54.292709 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:54.292371 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce82c78-f9wrj" event={"ID":"89a64fcf-55f5-4d7f-8a79-f97d61b582d1","Type":"ContainerDied","Data":"c8ad86110fc66df2ee6f01493035a478ad7d68a82d6910414ca9dd893bbfe5cc"} Oct 30 22:10:54.571971 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:54.571871 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-691598-n2d4g"] Oct 30 22:10:54.574627 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:54.574596 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-691598-n2d4g"] Oct 30 22:10:55.434370 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:55.434345 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce82c78-f9wrj" Oct 30 22:10:55.532601 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:55.532567 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89a64fcf-55f5-4d7f-8a79-f97d61b582d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89a64fcf-55f5-4d7f-8a79-f97d61b582d1-maas-api-xks-ca\") pod \"89a64fcf-55f5-4d7f-8a79-f97d61b582d1\" (UID: \"89a64fcf-55f5-4d7f-8a79-f97d61b582d1\") " Oct 30 22:10:55.532771 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:55.532630 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89a64fcf-55f5-4d7f-8a79-f97d61b582d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89a64fcf-55f5-4d7f-8a79-f97d61b582d1-maas-api-openshift-ca\") pod \"89a64fcf-55f5-4d7f-8a79-f97d61b582d1\" (UID: \"89a64fcf-55f5-4d7f-8a79-f97d61b582d1\") " Oct 30 22:10:55.532850 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:55.532799 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89a64fcf-55f5-4d7f-8a79-f97d61b582d1-kube-api-access-mdls8\" (UniqueName: \"kubernetes.io/projected/89a64fcf-55f5-4d7f-8a79-f97d61b582d1-kube-api-access-mdls8\") pod \"89a64fcf-55f5-4d7f-8a79-f97d61b582d1\" (UID: \"89a64fcf-55f5-4d7f-8a79-f97d61b582d1\") " Oct 30 22:10:55.533041 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:55.533014 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89a64fcf-55f5-4d7f-8a79-f97d61b582d1-maas-api-openshift-ca" pod "89a64fcf-55f5-4d7f-8a79-f97d61b582d1" (UID: "89a64fcf-55f5-4d7f-8a79-f97d61b582d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:10:55.533206 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:55.533186 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89a64fcf-55f5-4d7f-8a79-f97d61b582d1-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:55.534710 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:55.534685 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89a64fcf-55f5-4d7f-8a79-f97d61b582d1-maas-api-xks-ca" pod "89a64fcf-55f5-4d7f-8a79-f97d61b582d1" (UID: "89a64fcf-55f5-4d7f-8a79-f97d61b582d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:10:55.534830 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:55.534811 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89a64fcf-55f5-4d7f-8a79-f97d61b582d1-kube-api-access-mdls8" pod "89a64fcf-55f5-4d7f-8a79-f97d61b582d1" (UID: "89a64fcf-55f5-4d7f-8a79-f97d61b582d1"). InnerVolumeSpecName "kube-api-access-mdls8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:55.634493 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:55.634396 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdls8\" (UniqueName: \"kubernetes.io/projected/89a64fcf-55f5-4d7f-8a79-f97d61b582d1-kube-api-access-mdls8\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:55.634493 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:55.634439 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89a64fcf-55f5-4d7f-8a79-f97d61b582d1-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:55.867977 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:55.867937 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd2a94-wgrgm"] Oct 30 22:10:55.872147 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:55.872122 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd2a94-wgrgm"] Oct 30 22:10:56.305372 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:56.305338 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce82c78-f9wrj" Oct 30 22:10:56.305372 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:56.305367 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce82c78-f9wrj" event={"ID":"89a64fcf-55f5-4d7f-8a79-f97d61b582d1","Type":"ContainerDied","Data":"03888395dffe87a73c4d1532c36b5f208cd683e0807e0648428fc5f37cbdc8cf"} Oct 30 22:10:56.305597 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:56.305411 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03888395dffe87a73c4d1532c36b5f208cd683e0807e0648428fc5f37cbdc8cf" Oct 30 22:10:56.417719 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:56.417685 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75d93cb9-cdbe-4756-9350-0a74554c7bf5" path="/var/lib/kubelet/pods/75d93cb9-cdbe-4756-9350-0a74554c7bf5/volumes" Oct 30 22:10:56.418170 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:56.418154 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ded0f467-f8cb-4b8e-89e8-fbf29cc3189f" path="/var/lib/kubelet/pods/ded0f467-f8cb-4b8e-89e8-fbf29cc3189f/volumes" Oct 30 22:10:57.158193 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:57.158163 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1de75a-pfz4g"] Oct 30 22:10:57.164063 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:57.164039 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1de75a-pfz4g"] Oct 30 22:10:58.415769 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:10:58.415734 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b358bd3e-1b9a-4f5a-8c13-6dbb806d4874" path="/var/lib/kubelet/pods/b358bd3e-1b9a-4f5a-8c13-6dbb806d4874/volumes" Oct 30 22:11:00.144535 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:11:00.144496 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:11:04.254572 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:04.254540 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ea1f0b-trfrr"] Oct 30 22:11:04.255018 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:04.254982 5748 cpu_manager.go:414] "containerMap: removing container" podUID="89a64fcf-55f5-4d7f-8a79-f97d61b582d1" containerName="revoke-keys" Oct 30 22:11:04.255018 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:04.254994 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89a64fcf-55f5-4d7f-8a79-f97d61b582d1" containerName="revoke-keys" Oct 30 22:11:04.255109 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:04.255096 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="89a64fcf-55f5-4d7f-8a79-f97d61b582d1" containerName="revoke-keys" Oct 30 22:11:04.258479 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:04.258457 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ea1f0b-trfrr" Oct 30 22:11:04.313018 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:04.312994 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:11:04.320356 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:04.320332 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:11:04.363670 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:04.363641 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:11:04.367141 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:04.367118 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ea1f0b-trfrr"] Oct 30 22:11:04.370322 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:04.370300 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 22:11:04.424071 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:04.424030 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8addc976-b52c-442a-ba92-fe4f7f3be00a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ea1f0b-trfrr\" (UID: \"8addc976-b52c-442a-ba92-fe4f7f3be00a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ea1f0b-trfrr" Oct 30 22:11:04.424276 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:04.424111 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8addc976-b52c-442a-ba92-fe4f7f3be00a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ea1f0b-trfrr\" (UID: \"8addc976-b52c-442a-ba92-fe4f7f3be00a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ea1f0b-trfrr" Oct 30 22:11:04.424276 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:04.424150 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2jx8\" (UniqueName: \"kubernetes.io/projected/8addc976-b52c-442a-ba92-fe4f7f3be00a-kube-api-access-g2jx8\") pod \"maas-api-revoke-keys-e2e-inf-b-ea1f0b-trfrr\" (UID: \"8addc976-b52c-442a-ba92-fe4f7f3be00a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ea1f0b-trfrr" Oct 30 22:11:04.525142 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:04.525051 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8addc976-b52c-442a-ba92-fe4f7f3be00a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ea1f0b-trfrr\" (UID: \"8addc976-b52c-442a-ba92-fe4f7f3be00a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ea1f0b-trfrr" Oct 30 22:11:04.525142 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:04.525117 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8addc976-b52c-442a-ba92-fe4f7f3be00a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ea1f0b-trfrr\" (UID: \"8addc976-b52c-442a-ba92-fe4f7f3be00a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ea1f0b-trfrr" Oct 30 22:11:04.525410 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:04.525147 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2jx8\" (UniqueName: \"kubernetes.io/projected/8addc976-b52c-442a-ba92-fe4f7f3be00a-kube-api-access-g2jx8\") pod \"maas-api-revoke-keys-e2e-inf-b-ea1f0b-trfrr\" (UID: \"8addc976-b52c-442a-ba92-fe4f7f3be00a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ea1f0b-trfrr" Oct 30 22:11:04.526121 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:04.526087 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8addc976-b52c-442a-ba92-fe4f7f3be00a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ea1f0b-trfrr\" (UID: \"8addc976-b52c-442a-ba92-fe4f7f3be00a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ea1f0b-trfrr" Oct 30 22:11:04.528278 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:04.528249 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8addc976-b52c-442a-ba92-fe4f7f3be00a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ea1f0b-trfrr\" (UID: \"8addc976-b52c-442a-ba92-fe4f7f3be00a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ea1f0b-trfrr" Oct 30 22:11:04.587025 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:04.586994 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2jx8\" (UniqueName: \"kubernetes.io/projected/8addc976-b52c-442a-ba92-fe4f7f3be00a-kube-api-access-g2jx8\") pod \"maas-api-revoke-keys-e2e-inf-b-ea1f0b-trfrr\" (UID: \"8addc976-b52c-442a-ba92-fe4f7f3be00a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ea1f0b-trfrr" Oct 30 22:11:04.871268 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:04.871180 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ea1f0b-trfrr" Oct 30 22:11:05.251828 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:05.251796 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ea1f0b-trfrr"] Oct 30 22:11:05.254365 ip-10-0-143-61 kubenswrapper[5748]: W1030 22:11:05.254336 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8addc976_b52c_442a_ba92_fe4f7f3be00a.slice/crio-efc2d9aeb381d9a82634efb91f761c3765d7d6b8ba2474b9dd4ab0fdfd27f720 WatchSource:0}: Error finding container efc2d9aeb381d9a82634efb91f761c3765d7d6b8ba2474b9dd4ab0fdfd27f720: Status 404 returned error can't find the container with id efc2d9aeb381d9a82634efb91f761c3765d7d6b8ba2474b9dd4ab0fdfd27f720 Oct 30 22:11:05.359920 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:05.359877 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ea1f0b-trfrr" event={"ID":"8addc976-b52c-442a-ba92-fe4f7f3be00a","Type":"ContainerStarted","Data":"efc2d9aeb381d9a82634efb91f761c3765d7d6b8ba2474b9dd4ab0fdfd27f720"} Oct 30 22:11:06.367930 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:06.367839 5748 generic.go:356] "Generic (PLEG): container finished" podID="8addc976-b52c-442a-ba92-fe4f7f3be00a" containerID="c399ddfb9d2ed1570bd8cce9b78676e8deeb686d751660e29c561fd19b421336" exitCode=0 Oct 30 22:11:06.367930 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:06.367891 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ea1f0b-trfrr" event={"ID":"8addc976-b52c-442a-ba92-fe4f7f3be00a","Type":"ContainerDied","Data":"c399ddfb9d2ed1570bd8cce9b78676e8deeb686d751660e29c561fd19b421336"} Oct 30 22:11:07.514322 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:07.514297 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ea1f0b-trfrr" Oct 30 22:11:07.658557 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:07.658527 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8addc976-b52c-442a-ba92-fe4f7f3be00a-kube-api-access-g2jx8\" (UniqueName: \"kubernetes.io/projected/8addc976-b52c-442a-ba92-fe4f7f3be00a-kube-api-access-g2jx8\") pod \"8addc976-b52c-442a-ba92-fe4f7f3be00a\" (UID: \"8addc976-b52c-442a-ba92-fe4f7f3be00a\") " Oct 30 22:11:07.658750 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:07.658597 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8addc976-b52c-442a-ba92-fe4f7f3be00a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8addc976-b52c-442a-ba92-fe4f7f3be00a-maas-api-openshift-ca\") pod \"8addc976-b52c-442a-ba92-fe4f7f3be00a\" (UID: \"8addc976-b52c-442a-ba92-fe4f7f3be00a\") " Oct 30 22:11:07.658750 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:07.658655 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8addc976-b52c-442a-ba92-fe4f7f3be00a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8addc976-b52c-442a-ba92-fe4f7f3be00a-maas-api-xks-ca\") pod \"8addc976-b52c-442a-ba92-fe4f7f3be00a\" (UID: \"8addc976-b52c-442a-ba92-fe4f7f3be00a\") " Oct 30 22:11:07.659034 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:07.659012 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8addc976-b52c-442a-ba92-fe4f7f3be00a-maas-api-openshift-ca" pod "8addc976-b52c-442a-ba92-fe4f7f3be00a" (UID: "8addc976-b52c-442a-ba92-fe4f7f3be00a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:11:07.660838 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:07.660806 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8addc976-b52c-442a-ba92-fe4f7f3be00a-maas-api-xks-ca" pod "8addc976-b52c-442a-ba92-fe4f7f3be00a" (UID: "8addc976-b52c-442a-ba92-fe4f7f3be00a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:11:07.660959 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:07.660831 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8addc976-b52c-442a-ba92-fe4f7f3be00a-kube-api-access-g2jx8" pod "8addc976-b52c-442a-ba92-fe4f7f3be00a" (UID: "8addc976-b52c-442a-ba92-fe4f7f3be00a"). InnerVolumeSpecName "kube-api-access-g2jx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:07.759959 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:07.759924 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2jx8\" (UniqueName: \"kubernetes.io/projected/8addc976-b52c-442a-ba92-fe4f7f3be00a-kube-api-access-g2jx8\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:07.759959 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:07.759958 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8addc976-b52c-442a-ba92-fe4f7f3be00a-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:07.760156 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:07.759974 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8addc976-b52c-442a-ba92-fe4f7f3be00a-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:08.380374 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:08.380345 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ea1f0b-trfrr" Oct 30 22:11:08.380562 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:08.380381 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ea1f0b-trfrr" event={"ID":"8addc976-b52c-442a-ba92-fe4f7f3be00a","Type":"ContainerDied","Data":"efc2d9aeb381d9a82634efb91f761c3765d7d6b8ba2474b9dd4ab0fdfd27f720"} Oct 30 22:11:08.380562 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:08.380429 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="efc2d9aeb381d9a82634efb91f761c3765d7d6b8ba2474b9dd4ab0fdfd27f720" Oct 30 22:11:08.881804 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:08.881767 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6ea374-gbn2r"] Oct 30 22:11:08.885255 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:08.885227 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6ea374-gbn2r"] Oct 30 22:11:10.416361 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:10.416323 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="665fa115-464b-4259-9f81-a2907f42db04" path="/var/lib/kubelet/pods/665fa115-464b-4259-9f81-a2907f42db04/volumes" Oct 30 22:11:16.511420 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:16.511379 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c"] Oct 30 22:11:16.511988 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:16.511601 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c" podUID="f74b9971-1030-4e52-8f27-6139e51d8a75" containerName="istio-proxy" containerID="cri-o://f73e7169910532cb88013316ae1c816310f7105e2b22a23741ded1e2dc587edb" gracePeriod=30 Oct 30 22:11:17.651372 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:17.651330 5748 patch_prober.go:29] interesting pod/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:11:17.651888 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:17.651394 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c" podUID="f74b9971-1030-4e52-8f27-6139e51d8a75" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:11:21.769225 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:21.769198 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c" Oct 30 22:11:21.907245 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:21.907209 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f74b9971-1030-4e52-8f27-6139e51d8a75-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f74b9971-1030-4e52-8f27-6139e51d8a75-istio-data\") pod \"f74b9971-1030-4e52-8f27-6139e51d8a75\" (UID: \"f74b9971-1030-4e52-8f27-6139e51d8a75\") " Oct 30 22:11:21.907435 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:21.907300 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f74b9971-1030-4e52-8f27-6139e51d8a75-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f74b9971-1030-4e52-8f27-6139e51d8a75-workload-certs\") pod \"f74b9971-1030-4e52-8f27-6139e51d8a75\" (UID: \"f74b9971-1030-4e52-8f27-6139e51d8a75\") " Oct 30 22:11:21.907435 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:21.907346 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f74b9971-1030-4e52-8f27-6139e51d8a75-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f74b9971-1030-4e52-8f27-6139e51d8a75-credential-socket\") pod \"f74b9971-1030-4e52-8f27-6139e51d8a75\" (UID: \"f74b9971-1030-4e52-8f27-6139e51d8a75\") " Oct 30 22:11:21.907435 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:21.907381 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f74b9971-1030-4e52-8f27-6139e51d8a75-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f74b9971-1030-4e52-8f27-6139e51d8a75-workload-socket\") pod \"f74b9971-1030-4e52-8f27-6139e51d8a75\" (UID: \"f74b9971-1030-4e52-8f27-6139e51d8a75\") " Oct 30 22:11:21.907624 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:21.907461 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f74b9971-1030-4e52-8f27-6139e51d8a75-istio-token\" (UniqueName: \"kubernetes.io/projected/f74b9971-1030-4e52-8f27-6139e51d8a75-istio-token\") pod \"f74b9971-1030-4e52-8f27-6139e51d8a75\" (UID: \"f74b9971-1030-4e52-8f27-6139e51d8a75\") " Oct 30 22:11:21.907624 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:21.907493 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f74b9971-1030-4e52-8f27-6139e51d8a75-istio-data" pod "f74b9971-1030-4e52-8f27-6139e51d8a75" (UID: "f74b9971-1030-4e52-8f27-6139e51d8a75"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:21.907624 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:21.907537 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f74b9971-1030-4e52-8f27-6139e51d8a75-kube-api-access-mp7cz\" (UniqueName: \"kubernetes.io/projected/f74b9971-1030-4e52-8f27-6139e51d8a75-kube-api-access-mp7cz\") pod \"f74b9971-1030-4e52-8f27-6139e51d8a75\" (UID: \"f74b9971-1030-4e52-8f27-6139e51d8a75\") " Oct 30 22:11:21.907624 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:21.907572 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f74b9971-1030-4e52-8f27-6139e51d8a75-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f74b9971-1030-4e52-8f27-6139e51d8a75-istio-podinfo\") pod \"f74b9971-1030-4e52-8f27-6139e51d8a75\" (UID: \"f74b9971-1030-4e52-8f27-6139e51d8a75\") " Oct 30 22:11:21.907624 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:21.907610 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f74b9971-1030-4e52-8f27-6139e51d8a75-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f74b9971-1030-4e52-8f27-6139e51d8a75-istiod-ca-cert\") pod \"f74b9971-1030-4e52-8f27-6139e51d8a75\" (UID: \"f74b9971-1030-4e52-8f27-6139e51d8a75\") " Oct 30 22:11:21.907883 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:21.907606 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f74b9971-1030-4e52-8f27-6139e51d8a75-workload-certs" pod "f74b9971-1030-4e52-8f27-6139e51d8a75" (UID: "f74b9971-1030-4e52-8f27-6139e51d8a75"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:21.907883 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:21.907642 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f74b9971-1030-4e52-8f27-6139e51d8a75-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f74b9971-1030-4e52-8f27-6139e51d8a75-istio-envoy\") pod \"f74b9971-1030-4e52-8f27-6139e51d8a75\" (UID: \"f74b9971-1030-4e52-8f27-6139e51d8a75\") " Oct 30 22:11:21.907883 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:21.907684 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f74b9971-1030-4e52-8f27-6139e51d8a75-credential-socket" pod "f74b9971-1030-4e52-8f27-6139e51d8a75" (UID: "f74b9971-1030-4e52-8f27-6139e51d8a75"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:21.907883 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:21.907718 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f74b9971-1030-4e52-8f27-6139e51d8a75-workload-socket" pod "f74b9971-1030-4e52-8f27-6139e51d8a75" (UID: "f74b9971-1030-4e52-8f27-6139e51d8a75"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:21.908192 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:21.908035 5748 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f74b9971-1030-4e52-8f27-6139e51d8a75-workload-certs\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:21.908192 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:21.908059 5748 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f74b9971-1030-4e52-8f27-6139e51d8a75-credential-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:21.908192 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:21.908073 5748 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f74b9971-1030-4e52-8f27-6139e51d8a75-workload-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:21.908192 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:21.908086 5748 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f74b9971-1030-4e52-8f27-6139e51d8a75-istio-data\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:21.908192 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:21.908039 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f74b9971-1030-4e52-8f27-6139e51d8a75-istiod-ca-cert" pod "f74b9971-1030-4e52-8f27-6139e51d8a75" (UID: "f74b9971-1030-4e52-8f27-6139e51d8a75"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:11:21.910452 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:21.910426 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f74b9971-1030-4e52-8f27-6139e51d8a75-istio-envoy" pod "f74b9971-1030-4e52-8f27-6139e51d8a75" (UID: "f74b9971-1030-4e52-8f27-6139e51d8a75"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:21.910607 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:21.910515 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f74b9971-1030-4e52-8f27-6139e51d8a75-istio-token" pod "f74b9971-1030-4e52-8f27-6139e51d8a75" (UID: "f74b9971-1030-4e52-8f27-6139e51d8a75"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:21.910607 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:21.910532 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f74b9971-1030-4e52-8f27-6139e51d8a75-kube-api-access-mp7cz" pod "f74b9971-1030-4e52-8f27-6139e51d8a75" (UID: "f74b9971-1030-4e52-8f27-6139e51d8a75"). InnerVolumeSpecName "kube-api-access-mp7cz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:21.910752 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:21.910627 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f74b9971-1030-4e52-8f27-6139e51d8a75-istio-podinfo" pod "f74b9971-1030-4e52-8f27-6139e51d8a75" (UID: "f74b9971-1030-4e52-8f27-6139e51d8a75"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:11:22.009219 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:22.009181 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp7cz\" (UniqueName: \"kubernetes.io/projected/f74b9971-1030-4e52-8f27-6139e51d8a75-kube-api-access-mp7cz\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:22.009219 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:22.009214 5748 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f74b9971-1030-4e52-8f27-6139e51d8a75-istio-podinfo\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:22.009431 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:22.009230 5748 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f74b9971-1030-4e52-8f27-6139e51d8a75-istiod-ca-cert\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:22.009431 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:22.009244 5748 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f74b9971-1030-4e52-8f27-6139e51d8a75-istio-envoy\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:22.009431 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:22.009256 5748 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f74b9971-1030-4e52-8f27-6139e51d8a75-istio-token\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:22.486039 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:22.486004 5748 generic.go:356] "Generic (PLEG): container finished" podID="f74b9971-1030-4e52-8f27-6139e51d8a75" containerID="f73e7169910532cb88013316ae1c816310f7105e2b22a23741ded1e2dc587edb" exitCode=0 Oct 30 22:11:22.486209 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:22.486055 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c" event={"ID":"f74b9971-1030-4e52-8f27-6139e51d8a75","Type":"ContainerDied","Data":"f73e7169910532cb88013316ae1c816310f7105e2b22a23741ded1e2dc587edb"} Oct 30 22:11:22.486209 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:22.486068 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c" Oct 30 22:11:22.486209 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:22.486098 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c" event={"ID":"f74b9971-1030-4e52-8f27-6139e51d8a75","Type":"ContainerDied","Data":"87b343f07d13bd8787b00e1002851922e1e94b1f3ba72240257a2f77dd77d653"} Oct 30 22:11:22.486209 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:22.486119 5748 scope.go:122] "RemoveContainer" containerID="f73e7169910532cb88013316ae1c816310f7105e2b22a23741ded1e2dc587edb" Oct 30 22:11:22.496364 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:22.496333 5748 scope.go:122] "RemoveContainer" containerID="f73e7169910532cb88013316ae1c816310f7105e2b22a23741ded1e2dc587edb" Oct 30 22:11:22.496708 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:11:22.496679 5748 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f73e7169910532cb88013316ae1c816310f7105e2b22a23741ded1e2dc587edb\": container with ID starting with f73e7169910532cb88013316ae1c816310f7105e2b22a23741ded1e2dc587edb not found: ID does not exist" containerID="f73e7169910532cb88013316ae1c816310f7105e2b22a23741ded1e2dc587edb" Oct 30 22:11:22.496784 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:22.496720 5748 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f73e7169910532cb88013316ae1c816310f7105e2b22a23741ded1e2dc587edb"} err="failed to get container status \"f73e7169910532cb88013316ae1c816310f7105e2b22a23741ded1e2dc587edb\": rpc error: code = NotFound desc = could not find container \"f73e7169910532cb88013316ae1c816310f7105e2b22a23741ded1e2dc587edb\": container with ID starting with f73e7169910532cb88013316ae1c816310f7105e2b22a23741ded1e2dc587edb not found: ID does not exist" Oct 30 22:11:22.704605 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:22.704522 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c"] Oct 30 22:11:22.708572 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:22.708542 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-s2p6c"] Oct 30 22:11:24.416471 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:24.416439 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f74b9971-1030-4e52-8f27-6139e51d8a75" path="/var/lib/kubelet/pods/f74b9971-1030-4e52-8f27-6139e51d8a75/volumes" Oct 30 22:11:27.993162 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:27.993120 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4307febc-nxghv"] Oct 30 22:11:27.993688 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:27.993556 5748 cpu_manager.go:414] "containerMap: removing container" podUID="8addc976-b52c-442a-ba92-fe4f7f3be00a" containerName="revoke-keys" Oct 30 22:11:27.993688 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:27.993568 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8addc976-b52c-442a-ba92-fe4f7f3be00a" containerName="revoke-keys" Oct 30 22:11:27.993688 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:27.993592 5748 cpu_manager.go:414] "containerMap: removing container" podUID="f74b9971-1030-4e52-8f27-6139e51d8a75" containerName="istio-proxy" Oct 30 22:11:27.993688 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:27.993597 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f74b9971-1030-4e52-8f27-6139e51d8a75" containerName="istio-proxy" Oct 30 22:11:27.993688 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:27.993687 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="8addc976-b52c-442a-ba92-fe4f7f3be00a" containerName="revoke-keys" Oct 30 22:11:27.994007 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:27.993698 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="f74b9971-1030-4e52-8f27-6139e51d8a75" containerName="istio-proxy" Oct 30 22:11:27.998512 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:27.998489 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4307febc-nxghv" Oct 30 22:11:28.055037 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:28.054997 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4307febc-nxghv"] Oct 30 22:11:28.073031 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:28.073001 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f77651a-b47a-46f8-9d33-a6baa49bc9db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4307febc-nxghv\" (UID: \"4f77651a-b47a-46f8-9d33-a6baa49bc9db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4307febc-nxghv" Oct 30 22:11:28.073031 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:28.073033 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vqb6\" (UniqueName: \"kubernetes.io/projected/4f77651a-b47a-46f8-9d33-a6baa49bc9db-kube-api-access-5vqb6\") pod \"maas-api-revoke-sub-4307febc-nxghv\" (UID: \"4f77651a-b47a-46f8-9d33-a6baa49bc9db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4307febc-nxghv" Oct 30 22:11:28.073254 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:28.073077 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f77651a-b47a-46f8-9d33-a6baa49bc9db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4307febc-nxghv\" (UID: \"4f77651a-b47a-46f8-9d33-a6baa49bc9db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4307febc-nxghv" Oct 30 22:11:28.102727 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:28.102697 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:11:28.102727 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:28.102721 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:11:28.102974 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:28.102752 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:11:28.102974 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:28.102809 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 22:11:28.174223 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:28.174189 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f77651a-b47a-46f8-9d33-a6baa49bc9db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4307febc-nxghv\" (UID: \"4f77651a-b47a-46f8-9d33-a6baa49bc9db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4307febc-nxghv" Oct 30 22:11:28.174223 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:28.174222 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vqb6\" (UniqueName: \"kubernetes.io/projected/4f77651a-b47a-46f8-9d33-a6baa49bc9db-kube-api-access-5vqb6\") pod \"maas-api-revoke-sub-4307febc-nxghv\" (UID: \"4f77651a-b47a-46f8-9d33-a6baa49bc9db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4307febc-nxghv" Oct 30 22:11:28.174455 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:28.174260 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f77651a-b47a-46f8-9d33-a6baa49bc9db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4307febc-nxghv\" (UID: \"4f77651a-b47a-46f8-9d33-a6baa49bc9db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4307febc-nxghv" Oct 30 22:11:28.174817 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:28.174795 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f77651a-b47a-46f8-9d33-a6baa49bc9db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4307febc-nxghv\" (UID: \"4f77651a-b47a-46f8-9d33-a6baa49bc9db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4307febc-nxghv" Oct 30 22:11:28.176612 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:28.176584 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f77651a-b47a-46f8-9d33-a6baa49bc9db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4307febc-nxghv\" (UID: \"4f77651a-b47a-46f8-9d33-a6baa49bc9db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4307febc-nxghv" Oct 30 22:11:28.236060 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:28.236030 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vqb6\" (UniqueName: \"kubernetes.io/projected/4f77651a-b47a-46f8-9d33-a6baa49bc9db-kube-api-access-5vqb6\") pod \"maas-api-revoke-sub-4307febc-nxghv\" (UID: \"4f77651a-b47a-46f8-9d33-a6baa49bc9db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4307febc-nxghv" Oct 30 22:11:28.314730 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:28.314653 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4307febc-nxghv" Oct 30 22:11:28.687534 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:28.687509 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4307febc-nxghv"] Oct 30 22:11:28.689411 ip-10-0-143-61 kubenswrapper[5748]: W1030 22:11:28.689382 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f77651a_b47a_46f8_9d33_a6baa49bc9db.slice/crio-bdce9322b46597c188eb72d85c0c205a700982c518cb607f97db63e235105ad5 WatchSource:0}: Error finding container bdce9322b46597c188eb72d85c0c205a700982c518cb607f97db63e235105ad5: Status 404 returned error can't find the container with id bdce9322b46597c188eb72d85c0c205a700982c518cb607f97db63e235105ad5 Oct 30 22:11:29.530492 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:29.530455 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4307febc-nxghv" event={"ID":"4f77651a-b47a-46f8-9d33-a6baa49bc9db","Type":"ContainerStarted","Data":"edcb20a0396a3fcf21de0c3fa4e274b991bb69c5fa69fd3dc9a1d73e71410380"} Oct 30 22:11:29.530959 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:29.530508 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4307febc-nxghv" event={"ID":"4f77651a-b47a-46f8-9d33-a6baa49bc9db","Type":"ContainerStarted","Data":"bdce9322b46597c188eb72d85c0c205a700982c518cb607f97db63e235105ad5"} Oct 30 22:11:29.649402 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:29.648602 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4307febc-nxghv" podStartSLOduration=2.648583427 podStartE2EDuration="2.648583427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:29.64814842 +0000 UTC m=+2873.390669357" watchObservedRunningTime="2026-10-30 22:11:29.648583427 +0000 UTC m=+2873.391104364" Oct 30 22:11:30.144304 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:11:30.144271 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:11:30.537912 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:30.537821 5748 generic.go:356] "Generic (PLEG): container finished" podID="4f77651a-b47a-46f8-9d33-a6baa49bc9db" containerID="edcb20a0396a3fcf21de0c3fa4e274b991bb69c5fa69fd3dc9a1d73e71410380" exitCode=0 Oct 30 22:11:30.537912 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:30.537870 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4307febc-nxghv" event={"ID":"4f77651a-b47a-46f8-9d33-a6baa49bc9db","Type":"ContainerDied","Data":"edcb20a0396a3fcf21de0c3fa4e274b991bb69c5fa69fd3dc9a1d73e71410380"} Oct 30 22:11:31.678664 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:31.678642 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4307febc-nxghv" Oct 30 22:11:31.811283 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:31.811189 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f77651a-b47a-46f8-9d33-a6baa49bc9db-kube-api-access-5vqb6\" (UniqueName: \"kubernetes.io/projected/4f77651a-b47a-46f8-9d33-a6baa49bc9db-kube-api-access-5vqb6\") pod \"4f77651a-b47a-46f8-9d33-a6baa49bc9db\" (UID: \"4f77651a-b47a-46f8-9d33-a6baa49bc9db\") " Oct 30 22:11:31.811283 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:31.811254 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f77651a-b47a-46f8-9d33-a6baa49bc9db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f77651a-b47a-46f8-9d33-a6baa49bc9db-maas-api-xks-ca\") pod \"4f77651a-b47a-46f8-9d33-a6baa49bc9db\" (UID: \"4f77651a-b47a-46f8-9d33-a6baa49bc9db\") " Oct 30 22:11:31.811488 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:31.811313 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f77651a-b47a-46f8-9d33-a6baa49bc9db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f77651a-b47a-46f8-9d33-a6baa49bc9db-maas-api-openshift-ca\") pod \"4f77651a-b47a-46f8-9d33-a6baa49bc9db\" (UID: \"4f77651a-b47a-46f8-9d33-a6baa49bc9db\") " Oct 30 22:11:31.811675 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:31.811652 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f77651a-b47a-46f8-9d33-a6baa49bc9db-maas-api-openshift-ca" pod "4f77651a-b47a-46f8-9d33-a6baa49bc9db" (UID: "4f77651a-b47a-46f8-9d33-a6baa49bc9db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:11:31.811770 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:31.811752 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f77651a-b47a-46f8-9d33-a6baa49bc9db-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:31.813241 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:31.813216 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f77651a-b47a-46f8-9d33-a6baa49bc9db-kube-api-access-5vqb6" pod "4f77651a-b47a-46f8-9d33-a6baa49bc9db" (UID: "4f77651a-b47a-46f8-9d33-a6baa49bc9db"). InnerVolumeSpecName "kube-api-access-5vqb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:31.813380 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:31.813362 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f77651a-b47a-46f8-9d33-a6baa49bc9db-maas-api-xks-ca" pod "4f77651a-b47a-46f8-9d33-a6baa49bc9db" (UID: "4f77651a-b47a-46f8-9d33-a6baa49bc9db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:11:31.912826 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:31.912798 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f77651a-b47a-46f8-9d33-a6baa49bc9db-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:31.912826 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:31.912823 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vqb6\" (UniqueName: \"kubernetes.io/projected/4f77651a-b47a-46f8-9d33-a6baa49bc9db-kube-api-access-5vqb6\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:32.550882 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:32.550845 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4307febc-nxghv" event={"ID":"4f77651a-b47a-46f8-9d33-a6baa49bc9db","Type":"ContainerDied","Data":"bdce9322b46597c188eb72d85c0c205a700982c518cb607f97db63e235105ad5"} Oct 30 22:11:32.551074 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:32.550919 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bdce9322b46597c188eb72d85c0c205a700982c518cb607f97db63e235105ad5" Oct 30 22:11:32.551074 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:32.550857 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4307febc-nxghv" Oct 30 22:11:35.404083 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:35.404051 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ea1f0b-trfrr"] Oct 30 22:11:35.454245 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:35.454214 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ea1f0b-trfrr"] Oct 30 22:11:36.415510 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:36.415474 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8addc976-b52c-442a-ba92-fe4f7f3be00a" path="/var/lib/kubelet/pods/8addc976-b52c-442a-ba92-fe4f7f3be00a/volumes" Oct 30 22:11:38.361235 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:38.361200 5748 scope.go:122] "RemoveContainer" containerID="fe83cc9d147468c556219123b06f6ae8ba24ce21f3b201f2efcf45b470e2896b" Oct 30 22:11:38.370790 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:38.370771 5748 scope.go:122] "RemoveContainer" containerID="52f65eb7588132cfe7b918958c0ef17ef09f9344fa0d54d4f2a8088700350dc8" Oct 30 22:11:38.379314 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:38.379295 5748 scope.go:122] "RemoveContainer" containerID="cbdaaa39def44633990a6b78e2a3c18b37dc37b8f37a78bc025c946099f84485" Oct 30 22:11:38.387390 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:38.387372 5748 scope.go:122] "RemoveContainer" containerID="5243e6663aee3382320cf32fff162004e5c2042e278384c08b1e333f804c16c2" Oct 30 22:11:38.395411 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:38.395392 5748 scope.go:122] "RemoveContainer" containerID="c88f3c0140646a4fbaa46b19401b696c9737cbf9f08a30a377df9378ba23d284" Oct 30 22:11:43.693097 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:43.693023 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c7d477-5nl7t"] Oct 30 22:11:43.693491 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:43.693477 5748 cpu_manager.go:414] "containerMap: removing container" podUID="4f77651a-b47a-46f8-9d33-a6baa49bc9db" containerName="revoke-keys" Oct 30 22:11:43.693560 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:43.693492 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f77651a-b47a-46f8-9d33-a6baa49bc9db" containerName="revoke-keys" Oct 30 22:11:43.693560 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:43.693556 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f77651a-b47a-46f8-9d33-a6baa49bc9db" containerName="revoke-keys" Oct 30 22:11:43.698631 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:43.698610 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c7d477-5nl7t" Oct 30 22:11:43.723822 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:43.723798 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ba5816c-29e8-4d05-b67b-1674503fb759-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-c7d477-5nl7t\" (UID: \"4ba5816c-29e8-4d05-b67b-1674503fb759\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c7d477-5nl7t" Oct 30 22:11:43.723950 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:43.723828 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ba5816c-29e8-4d05-b67b-1674503fb759-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-c7d477-5nl7t\" (UID: \"4ba5816c-29e8-4d05-b67b-1674503fb759\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c7d477-5nl7t" Oct 30 22:11:43.723950 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:43.723859 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvlqm\" (UniqueName: \"kubernetes.io/projected/4ba5816c-29e8-4d05-b67b-1674503fb759-kube-api-access-lvlqm\") pod \"maas-api-revoke-keys-e2e-worker-w5-c7d477-5nl7t\" (UID: \"4ba5816c-29e8-4d05-b67b-1674503fb759\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c7d477-5nl7t" Oct 30 22:11:43.756757 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:43.756732 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:11:43.757008 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:43.756976 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:11:43.807461 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:43.807429 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:11:43.807461 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:43.807458 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 22:11:43.809512 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:43.809493 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c7d477-5nl7t"] Oct 30 22:11:43.824671 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:43.824650 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ba5816c-29e8-4d05-b67b-1674503fb759-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-c7d477-5nl7t\" (UID: \"4ba5816c-29e8-4d05-b67b-1674503fb759\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c7d477-5nl7t" Oct 30 22:11:43.824795 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:43.824676 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ba5816c-29e8-4d05-b67b-1674503fb759-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-c7d477-5nl7t\" (UID: \"4ba5816c-29e8-4d05-b67b-1674503fb759\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c7d477-5nl7t" Oct 30 22:11:43.824795 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:43.824704 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvlqm\" (UniqueName: \"kubernetes.io/projected/4ba5816c-29e8-4d05-b67b-1674503fb759-kube-api-access-lvlqm\") pod \"maas-api-revoke-keys-e2e-worker-w5-c7d477-5nl7t\" (UID: \"4ba5816c-29e8-4d05-b67b-1674503fb759\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c7d477-5nl7t" Oct 30 22:11:43.825383 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:43.825359 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ba5816c-29e8-4d05-b67b-1674503fb759-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-c7d477-5nl7t\" (UID: \"4ba5816c-29e8-4d05-b67b-1674503fb759\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c7d477-5nl7t" Oct 30 22:11:43.827096 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:43.827076 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ba5816c-29e8-4d05-b67b-1674503fb759-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-c7d477-5nl7t\" (UID: \"4ba5816c-29e8-4d05-b67b-1674503fb759\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c7d477-5nl7t" Oct 30 22:11:43.893629 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:43.893584 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvlqm\" (UniqueName: \"kubernetes.io/projected/4ba5816c-29e8-4d05-b67b-1674503fb759-kube-api-access-lvlqm\") pod \"maas-api-revoke-keys-e2e-worker-w5-c7d477-5nl7t\" (UID: \"4ba5816c-29e8-4d05-b67b-1674503fb759\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c7d477-5nl7t" Oct 30 22:11:44.011315 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:44.011250 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c7d477-5nl7t" Oct 30 22:11:44.390637 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:44.390611 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c7d477-5nl7t"] Oct 30 22:11:44.392481 ip-10-0-143-61 kubenswrapper[5748]: W1030 22:11:44.392451 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ba5816c_29e8_4d05_b67b_1674503fb759.slice/crio-a31fce7c7222a24bad591cbd90dddb9d50867db6a10fba5c9e087bd4108ce35b WatchSource:0}: Error finding container a31fce7c7222a24bad591cbd90dddb9d50867db6a10fba5c9e087bd4108ce35b: Status 404 returned error can't find the container with id a31fce7c7222a24bad591cbd90dddb9d50867db6a10fba5c9e087bd4108ce35b Oct 30 22:11:44.620153 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:44.620117 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c7d477-5nl7t" event={"ID":"4ba5816c-29e8-4d05-b67b-1674503fb759","Type":"ContainerStarted","Data":"a31fce7c7222a24bad591cbd90dddb9d50867db6a10fba5c9e087bd4108ce35b"} Oct 30 22:11:45.627496 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:45.627406 5748 generic.go:356] "Generic (PLEG): container finished" podID="4ba5816c-29e8-4d05-b67b-1674503fb759" containerID="6fec63d1e512a80b49f4dd46c6959f7e2a3f3e56483ec045d45b1f36486e75f8" exitCode=0 Oct 30 22:11:45.627934 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:45.627492 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c7d477-5nl7t" event={"ID":"4ba5816c-29e8-4d05-b67b-1674503fb759","Type":"ContainerDied","Data":"6fec63d1e512a80b49f4dd46c6959f7e2a3f3e56483ec045d45b1f36486e75f8"} Oct 30 22:11:46.776041 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:46.776015 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c7d477-5nl7t" Oct 30 22:11:46.851388 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:46.851351 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ba5816c-29e8-4d05-b67b-1674503fb759-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ba5816c-29e8-4d05-b67b-1674503fb759-maas-api-openshift-ca\") pod \"4ba5816c-29e8-4d05-b67b-1674503fb759\" (UID: \"4ba5816c-29e8-4d05-b67b-1674503fb759\") " Oct 30 22:11:46.851547 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:46.851403 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ba5816c-29e8-4d05-b67b-1674503fb759-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ba5816c-29e8-4d05-b67b-1674503fb759-maas-api-xks-ca\") pod \"4ba5816c-29e8-4d05-b67b-1674503fb759\" (UID: \"4ba5816c-29e8-4d05-b67b-1674503fb759\") " Oct 30 22:11:46.851547 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:46.851479 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ba5816c-29e8-4d05-b67b-1674503fb759-kube-api-access-lvlqm\" (UniqueName: \"kubernetes.io/projected/4ba5816c-29e8-4d05-b67b-1674503fb759-kube-api-access-lvlqm\") pod \"4ba5816c-29e8-4d05-b67b-1674503fb759\" (UID: \"4ba5816c-29e8-4d05-b67b-1674503fb759\") " Oct 30 22:11:46.851712 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:46.851690 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ba5816c-29e8-4d05-b67b-1674503fb759-maas-api-openshift-ca" pod "4ba5816c-29e8-4d05-b67b-1674503fb759" (UID: "4ba5816c-29e8-4d05-b67b-1674503fb759"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:11:46.853685 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:46.853658 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ba5816c-29e8-4d05-b67b-1674503fb759-maas-api-xks-ca" pod "4ba5816c-29e8-4d05-b67b-1674503fb759" (UID: "4ba5816c-29e8-4d05-b67b-1674503fb759"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:11:46.853685 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:46.853663 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ba5816c-29e8-4d05-b67b-1674503fb759-kube-api-access-lvlqm" pod "4ba5816c-29e8-4d05-b67b-1674503fb759" (UID: "4ba5816c-29e8-4d05-b67b-1674503fb759"). InnerVolumeSpecName "kube-api-access-lvlqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:46.952999 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:46.952966 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ba5816c-29e8-4d05-b67b-1674503fb759-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:46.952999 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:46.952994 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ba5816c-29e8-4d05-b67b-1674503fb759-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:46.952999 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:46.953004 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvlqm\" (UniqueName: \"kubernetes.io/projected/4ba5816c-29e8-4d05-b67b-1674503fb759-kube-api-access-lvlqm\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:47.641697 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:47.641668 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c7d477-5nl7t" Oct 30 22:11:47.642002 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:47.641666 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c7d477-5nl7t" event={"ID":"4ba5816c-29e8-4d05-b67b-1674503fb759","Type":"ContainerDied","Data":"a31fce7c7222a24bad591cbd90dddb9d50867db6a10fba5c9e087bd4108ce35b"} Oct 30 22:11:47.642002 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:47.641788 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a31fce7c7222a24bad591cbd90dddb9d50867db6a10fba5c9e087bd4108ce35b" Oct 30 22:11:48.081011 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:48.080922 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-c7d477/distinct-w5-c7d477-kserve-554d7f974d-6zw6s"] Oct 30 22:11:48.102202 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:48.102167 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h"] Oct 30 22:11:54.980839 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:54.980806 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de309da5-rnj2v"] Oct 30 22:11:54.981307 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:54.981287 5748 cpu_manager.go:414] "containerMap: removing container" podUID="4ba5816c-29e8-4d05-b67b-1674503fb759" containerName="revoke-keys" Oct 30 22:11:54.981307 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:54.981306 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ba5816c-29e8-4d05-b67b-1674503fb759" containerName="revoke-keys" Oct 30 22:11:54.981443 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:54.981432 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ba5816c-29e8-4d05-b67b-1674503fb759" containerName="revoke-keys" Oct 30 22:11:54.984823 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:54.984807 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de309da5-rnj2v" Oct 30 22:11:55.029791 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:55.029752 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37f5380a-6450-448b-9f2e-39c0464dce94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de309da5-rnj2v\" (UID: \"37f5380a-6450-448b-9f2e-39c0464dce94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de309da5-rnj2v" Oct 30 22:11:55.029969 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:55.029913 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf9ld\" (UniqueName: \"kubernetes.io/projected/37f5380a-6450-448b-9f2e-39c0464dce94-kube-api-access-lf9ld\") pod \"maas-api-revoke-sub-de309da5-rnj2v\" (UID: \"37f5380a-6450-448b-9f2e-39c0464dce94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de309da5-rnj2v" Oct 30 22:11:55.030069 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:55.030049 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37f5380a-6450-448b-9f2e-39c0464dce94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de309da5-rnj2v\" (UID: \"37f5380a-6450-448b-9f2e-39c0464dce94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de309da5-rnj2v" Oct 30 22:11:55.039871 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:55.039844 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a0395a5-mjcrk"] Oct 30 22:11:55.043678 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:55.043644 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0395a5-mjcrk" Oct 30 22:11:55.088135 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:55.088099 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:11:55.088135 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:55.088108 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:11:55.088356 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:55.088189 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:11:55.093253 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:55.093233 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 22:11:55.095381 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:55.095361 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de309da5-rnj2v"] Oct 30 22:11:55.102515 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:55.102491 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a0395a5-mjcrk"] Oct 30 22:11:55.131530 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:55.131503 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37f5380a-6450-448b-9f2e-39c0464dce94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de309da5-rnj2v\" (UID: \"37f5380a-6450-448b-9f2e-39c0464dce94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de309da5-rnj2v" Oct 30 22:11:55.131675 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:55.131547 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37f5380a-6450-448b-9f2e-39c0464dce94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de309da5-rnj2v\" (UID: \"37f5380a-6450-448b-9f2e-39c0464dce94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de309da5-rnj2v" Oct 30 22:11:55.131675 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:55.131575 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe298e2d-f999-4757-afa1-efb5928ca9ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a0395a5-mjcrk\" (UID: \"fe298e2d-f999-4757-afa1-efb5928ca9ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0395a5-mjcrk" Oct 30 22:11:55.131675 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:55.131607 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe298e2d-f999-4757-afa1-efb5928ca9ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a0395a5-mjcrk\" (UID: \"fe298e2d-f999-4757-afa1-efb5928ca9ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0395a5-mjcrk" Oct 30 22:11:55.131675 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:55.131658 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf9ld\" (UniqueName: \"kubernetes.io/projected/37f5380a-6450-448b-9f2e-39c0464dce94-kube-api-access-lf9ld\") pod \"maas-api-revoke-sub-de309da5-rnj2v\" (UID: \"37f5380a-6450-448b-9f2e-39c0464dce94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de309da5-rnj2v" Oct 30 22:11:55.131848 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:55.131822 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmh7t\" (UniqueName: \"kubernetes.io/projected/fe298e2d-f999-4757-afa1-efb5928ca9ca-kube-api-access-tmh7t\") pod \"maas-api-revoke-sub-0a0395a5-mjcrk\" (UID: \"fe298e2d-f999-4757-afa1-efb5928ca9ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0395a5-mjcrk" Oct 30 22:11:55.132299 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:55.132278 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37f5380a-6450-448b-9f2e-39c0464dce94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de309da5-rnj2v\" (UID: \"37f5380a-6450-448b-9f2e-39c0464dce94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de309da5-rnj2v" Oct 30 22:11:55.133980 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:55.133962 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37f5380a-6450-448b-9f2e-39c0464dce94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de309da5-rnj2v\" (UID: \"37f5380a-6450-448b-9f2e-39c0464dce94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de309da5-rnj2v" Oct 30 22:11:55.193753 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:55.193715 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf9ld\" (UniqueName: \"kubernetes.io/projected/37f5380a-6450-448b-9f2e-39c0464dce94-kube-api-access-lf9ld\") pod \"maas-api-revoke-sub-de309da5-rnj2v\" (UID: \"37f5380a-6450-448b-9f2e-39c0464dce94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de309da5-rnj2v" Oct 30 22:11:55.232865 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:55.232790 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmh7t\" (UniqueName: \"kubernetes.io/projected/fe298e2d-f999-4757-afa1-efb5928ca9ca-kube-api-access-tmh7t\") pod \"maas-api-revoke-sub-0a0395a5-mjcrk\" (UID: \"fe298e2d-f999-4757-afa1-efb5928ca9ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0395a5-mjcrk" Oct 30 22:11:55.233027 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:55.232922 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe298e2d-f999-4757-afa1-efb5928ca9ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a0395a5-mjcrk\" (UID: \"fe298e2d-f999-4757-afa1-efb5928ca9ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0395a5-mjcrk" Oct 30 22:11:55.233027 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:55.232963 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe298e2d-f999-4757-afa1-efb5928ca9ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a0395a5-mjcrk\" (UID: \"fe298e2d-f999-4757-afa1-efb5928ca9ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0395a5-mjcrk" Oct 30 22:11:55.233524 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:55.233498 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe298e2d-f999-4757-afa1-efb5928ca9ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a0395a5-mjcrk\" (UID: \"fe298e2d-f999-4757-afa1-efb5928ca9ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0395a5-mjcrk" Oct 30 22:11:55.235261 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:55.235236 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe298e2d-f999-4757-afa1-efb5928ca9ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a0395a5-mjcrk\" (UID: \"fe298e2d-f999-4757-afa1-efb5928ca9ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0395a5-mjcrk" Oct 30 22:11:55.294447 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:55.294408 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmh7t\" (UniqueName: \"kubernetes.io/projected/fe298e2d-f999-4757-afa1-efb5928ca9ca-kube-api-access-tmh7t\") pod \"maas-api-revoke-sub-0a0395a5-mjcrk\" (UID: \"fe298e2d-f999-4757-afa1-efb5928ca9ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0395a5-mjcrk" Oct 30 22:11:55.297196 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:55.297177 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de309da5-rnj2v" Oct 30 22:11:55.359221 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:55.359184 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0395a5-mjcrk" Oct 30 22:11:55.675845 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:55.675680 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de309da5-rnj2v"] Oct 30 22:11:55.679508 ip-10-0-143-61 kubenswrapper[5748]: W1030 22:11:55.679473 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37f5380a_6450_448b_9f2e_39c0464dce94.slice/crio-cdf462b9d5840eb6fc72f10a0da4f5945c166a73531aef768473a3c80275e0fb WatchSource:0}: Error finding container cdf462b9d5840eb6fc72f10a0da4f5945c166a73531aef768473a3c80275e0fb: Status 404 returned error can't find the container with id cdf462b9d5840eb6fc72f10a0da4f5945c166a73531aef768473a3c80275e0fb Oct 30 22:11:55.687826 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:55.687798 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de309da5-rnj2v" event={"ID":"37f5380a-6450-448b-9f2e-39c0464dce94","Type":"ContainerStarted","Data":"cdf462b9d5840eb6fc72f10a0da4f5945c166a73531aef768473a3c80275e0fb"} Oct 30 22:11:55.805979 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:55.805954 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a0395a5-mjcrk"] Oct 30 22:11:55.807834 ip-10-0-143-61 kubenswrapper[5748]: W1030 22:11:55.807802 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe298e2d_f999_4757_afa1_efb5928ca9ca.slice/crio-8d20c860832b738d32d365c9f77957a891ff8335e94355108ab484c88bc043c7 WatchSource:0}: Error finding container 8d20c860832b738d32d365c9f77957a891ff8335e94355108ab484c88bc043c7: Status 404 returned error can't find the container with id 8d20c860832b738d32d365c9f77957a891ff8335e94355108ab484c88bc043c7 Oct 30 22:11:56.695383 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:56.695352 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de309da5-rnj2v" event={"ID":"37f5380a-6450-448b-9f2e-39c0464dce94","Type":"ContainerStarted","Data":"1a351ec478d1baee6660bc4a9fb1f789c5e5a3969cba661bb9b614fa7d260d7d"} Oct 30 22:11:56.696972 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:56.696940 5748 generic.go:356] "Generic (PLEG): container finished" podID="fe298e2d-f999-4757-afa1-efb5928ca9ca" containerID="c7b89a2ad4c2e45cbf94b550ba7ba26da8dbdfe4b008d936e657a16e2540c4f7" exitCode=0 Oct 30 22:11:56.697097 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:56.697022 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0395a5-mjcrk" event={"ID":"fe298e2d-f999-4757-afa1-efb5928ca9ca","Type":"ContainerDied","Data":"c7b89a2ad4c2e45cbf94b550ba7ba26da8dbdfe4b008d936e657a16e2540c4f7"} Oct 30 22:11:56.697097 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:56.697077 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0395a5-mjcrk" event={"ID":"fe298e2d-f999-4757-afa1-efb5928ca9ca","Type":"ContainerStarted","Data":"8d20c860832b738d32d365c9f77957a891ff8335e94355108ab484c88bc043c7"} Oct 30 22:11:56.818317 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:56.818266 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de309da5-rnj2v" podStartSLOduration=2.818253317 podStartE2EDuration="2.818253317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:56.815777109 +0000 UTC m=+2900.558298044" watchObservedRunningTime="2026-10-30 22:11:56.818253317 +0000 UTC m=+2900.560774265" Oct 30 22:11:57.703670 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:57.703639 5748 generic.go:356] "Generic (PLEG): container finished" podID="37f5380a-6450-448b-9f2e-39c0464dce94" containerID="1a351ec478d1baee6660bc4a9fb1f789c5e5a3969cba661bb9b614fa7d260d7d" exitCode=0 Oct 30 22:11:57.704101 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:57.703730 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de309da5-rnj2v" event={"ID":"37f5380a-6450-448b-9f2e-39c0464dce94","Type":"ContainerDied","Data":"1a351ec478d1baee6660bc4a9fb1f789c5e5a3969cba661bb9b614fa7d260d7d"} Oct 30 22:11:57.842299 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:57.842273 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0395a5-mjcrk" Oct 30 22:11:57.961986 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:57.961880 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe298e2d-f999-4757-afa1-efb5928ca9ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe298e2d-f999-4757-afa1-efb5928ca9ca-maas-api-openshift-ca\") pod \"fe298e2d-f999-4757-afa1-efb5928ca9ca\" (UID: \"fe298e2d-f999-4757-afa1-efb5928ca9ca\") " Oct 30 22:11:57.961986 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:57.961933 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe298e2d-f999-4757-afa1-efb5928ca9ca-kube-api-access-tmh7t\" (UniqueName: \"kubernetes.io/projected/fe298e2d-f999-4757-afa1-efb5928ca9ca-kube-api-access-tmh7t\") pod \"fe298e2d-f999-4757-afa1-efb5928ca9ca\" (UID: \"fe298e2d-f999-4757-afa1-efb5928ca9ca\") " Oct 30 22:11:57.962180 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:57.962073 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe298e2d-f999-4757-afa1-efb5928ca9ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe298e2d-f999-4757-afa1-efb5928ca9ca-maas-api-xks-ca\") pod \"fe298e2d-f999-4757-afa1-efb5928ca9ca\" (UID: \"fe298e2d-f999-4757-afa1-efb5928ca9ca\") " Oct 30 22:11:57.962302 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:57.962275 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe298e2d-f999-4757-afa1-efb5928ca9ca-maas-api-openshift-ca" pod "fe298e2d-f999-4757-afa1-efb5928ca9ca" (UID: "fe298e2d-f999-4757-afa1-efb5928ca9ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:11:57.962425 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:57.962410 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe298e2d-f999-4757-afa1-efb5928ca9ca-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:57.964055 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:57.964029 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe298e2d-f999-4757-afa1-efb5928ca9ca-kube-api-access-tmh7t" pod "fe298e2d-f999-4757-afa1-efb5928ca9ca" (UID: "fe298e2d-f999-4757-afa1-efb5928ca9ca"). InnerVolumeSpecName "kube-api-access-tmh7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:57.964175 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:57.964086 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe298e2d-f999-4757-afa1-efb5928ca9ca-maas-api-xks-ca" pod "fe298e2d-f999-4757-afa1-efb5928ca9ca" (UID: "fe298e2d-f999-4757-afa1-efb5928ca9ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:11:58.063860 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:58.063820 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmh7t\" (UniqueName: \"kubernetes.io/projected/fe298e2d-f999-4757-afa1-efb5928ca9ca-kube-api-access-tmh7t\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:58.063860 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:58.063857 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe298e2d-f999-4757-afa1-efb5928ca9ca-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:58.712825 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:58.712734 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0395a5-mjcrk" Oct 30 22:11:58.712825 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:58.712764 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0395a5-mjcrk" event={"ID":"fe298e2d-f999-4757-afa1-efb5928ca9ca","Type":"ContainerDied","Data":"8d20c860832b738d32d365c9f77957a891ff8335e94355108ab484c88bc043c7"} Oct 30 22:11:58.713327 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:58.712830 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d20c860832b738d32d365c9f77957a891ff8335e94355108ab484c88bc043c7" Oct 30 22:11:58.857016 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:58.856989 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de309da5-rnj2v" Oct 30 22:11:58.976056 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:58.975964 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37f5380a-6450-448b-9f2e-39c0464dce94-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37f5380a-6450-448b-9f2e-39c0464dce94-maas-api-xks-ca\") pod \"37f5380a-6450-448b-9f2e-39c0464dce94\" (UID: \"37f5380a-6450-448b-9f2e-39c0464dce94\") " Oct 30 22:11:58.976056 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:58.976034 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37f5380a-6450-448b-9f2e-39c0464dce94-kube-api-access-lf9ld\" (UniqueName: \"kubernetes.io/projected/37f5380a-6450-448b-9f2e-39c0464dce94-kube-api-access-lf9ld\") pod \"37f5380a-6450-448b-9f2e-39c0464dce94\" (UID: \"37f5380a-6450-448b-9f2e-39c0464dce94\") " Oct 30 22:11:58.976269 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:58.976223 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37f5380a-6450-448b-9f2e-39c0464dce94-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37f5380a-6450-448b-9f2e-39c0464dce94-maas-api-openshift-ca\") pod \"37f5380a-6450-448b-9f2e-39c0464dce94\" (UID: \"37f5380a-6450-448b-9f2e-39c0464dce94\") " Oct 30 22:11:58.976578 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:58.976545 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37f5380a-6450-448b-9f2e-39c0464dce94-maas-api-openshift-ca" pod "37f5380a-6450-448b-9f2e-39c0464dce94" (UID: "37f5380a-6450-448b-9f2e-39c0464dce94"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:11:58.978126 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:58.978097 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37f5380a-6450-448b-9f2e-39c0464dce94-maas-api-xks-ca" pod "37f5380a-6450-448b-9f2e-39c0464dce94" (UID: "37f5380a-6450-448b-9f2e-39c0464dce94"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:11:58.978242 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:58.978220 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37f5380a-6450-448b-9f2e-39c0464dce94-kube-api-access-lf9ld" pod "37f5380a-6450-448b-9f2e-39c0464dce94" (UID: "37f5380a-6450-448b-9f2e-39c0464dce94"). InnerVolumeSpecName "kube-api-access-lf9ld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:59.078152 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:59.078120 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37f5380a-6450-448b-9f2e-39c0464dce94-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:59.078152 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:59.078149 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37f5380a-6450-448b-9f2e-39c0464dce94-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:59.078152 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:59.078161 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf9ld\" (UniqueName: \"kubernetes.io/projected/37f5380a-6450-448b-9f2e-39c0464dce94-kube-api-access-lf9ld\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:59.396163 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:59.396124 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e6dcc0-wjsj7"] Oct 30 22:11:59.396659 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:59.396638 5748 cpu_manager.go:414] "containerMap: removing container" podUID="fe298e2d-f999-4757-afa1-efb5928ca9ca" containerName="revoke-keys" Oct 30 22:11:59.396777 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:59.396660 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe298e2d-f999-4757-afa1-efb5928ca9ca" containerName="revoke-keys" Oct 30 22:11:59.396777 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:59.396694 5748 cpu_manager.go:414] "containerMap: removing container" podUID="37f5380a-6450-448b-9f2e-39c0464dce94" containerName="revoke-keys" Oct 30 22:11:59.396777 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:59.396704 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37f5380a-6450-448b-9f2e-39c0464dce94" containerName="revoke-keys" Oct 30 22:11:59.396981 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:59.396824 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe298e2d-f999-4757-afa1-efb5928ca9ca" containerName="revoke-keys" Oct 30 22:11:59.396981 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:59.396842 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="37f5380a-6450-448b-9f2e-39c0464dce94" containerName="revoke-keys" Oct 30 22:11:59.400374 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:59.400354 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e6dcc0-wjsj7" Oct 30 22:11:59.478595 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:59.478563 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e6dcc0-wjsj7"] Oct 30 22:11:59.482075 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:59.482044 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt465\" (UniqueName: \"kubernetes.io/projected/b3d801e9-1bb2-4dc8-a8e0-c2d7806e1f88-kube-api-access-mt465\") pod \"maas-api-revoke-keys-e2e-inf-a-e6dcc0-wjsj7\" (UID: \"b3d801e9-1bb2-4dc8-a8e0-c2d7806e1f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e6dcc0-wjsj7" Oct 30 22:11:59.482218 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:59.482172 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3d801e9-1bb2-4dc8-a8e0-c2d7806e1f88-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e6dcc0-wjsj7\" (UID: \"b3d801e9-1bb2-4dc8-a8e0-c2d7806e1f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e6dcc0-wjsj7" Oct 30 22:11:59.482372 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:59.482340 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3d801e9-1bb2-4dc8-a8e0-c2d7806e1f88-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e6dcc0-wjsj7\" (UID: \"b3d801e9-1bb2-4dc8-a8e0-c2d7806e1f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e6dcc0-wjsj7" Oct 30 22:11:59.583342 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:59.583311 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3d801e9-1bb2-4dc8-a8e0-c2d7806e1f88-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e6dcc0-wjsj7\" (UID: \"b3d801e9-1bb2-4dc8-a8e0-c2d7806e1f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e6dcc0-wjsj7" Oct 30 22:11:59.583511 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:59.583362 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt465\" (UniqueName: \"kubernetes.io/projected/b3d801e9-1bb2-4dc8-a8e0-c2d7806e1f88-kube-api-access-mt465\") pod \"maas-api-revoke-keys-e2e-inf-a-e6dcc0-wjsj7\" (UID: \"b3d801e9-1bb2-4dc8-a8e0-c2d7806e1f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e6dcc0-wjsj7" Oct 30 22:11:59.583511 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:59.583421 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3d801e9-1bb2-4dc8-a8e0-c2d7806e1f88-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e6dcc0-wjsj7\" (UID: \"b3d801e9-1bb2-4dc8-a8e0-c2d7806e1f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e6dcc0-wjsj7" Oct 30 22:11:59.584016 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:59.583990 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3d801e9-1bb2-4dc8-a8e0-c2d7806e1f88-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e6dcc0-wjsj7\" (UID: \"b3d801e9-1bb2-4dc8-a8e0-c2d7806e1f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e6dcc0-wjsj7" Oct 30 22:11:59.585747 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:59.585728 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3d801e9-1bb2-4dc8-a8e0-c2d7806e1f88-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e6dcc0-wjsj7\" (UID: \"b3d801e9-1bb2-4dc8-a8e0-c2d7806e1f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e6dcc0-wjsj7" Oct 30 22:11:59.649241 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:59.649171 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt465\" (UniqueName: \"kubernetes.io/projected/b3d801e9-1bb2-4dc8-a8e0-c2d7806e1f88-kube-api-access-mt465\") pod \"maas-api-revoke-keys-e2e-inf-a-e6dcc0-wjsj7\" (UID: \"b3d801e9-1bb2-4dc8-a8e0-c2d7806e1f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e6dcc0-wjsj7" Oct 30 22:11:59.715013 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:59.714978 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e6dcc0-wjsj7" Oct 30 22:11:59.720620 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:59.720599 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de309da5-rnj2v" Oct 30 22:11:59.720730 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:59.720628 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de309da5-rnj2v" event={"ID":"37f5380a-6450-448b-9f2e-39c0464dce94","Type":"ContainerDied","Data":"cdf462b9d5840eb6fc72f10a0da4f5945c166a73531aef768473a3c80275e0fb"} Oct 30 22:11:59.720730 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:11:59.720672 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cdf462b9d5840eb6fc72f10a0da4f5945c166a73531aef768473a3c80275e0fb" Oct 30 22:12:00.104655 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:00.104628 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e6dcc0-wjsj7"] Oct 30 22:12:00.106609 ip-10-0-143-61 kubenswrapper[5748]: W1030 22:12:00.106581 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3d801e9_1bb2_4dc8_a8e0_c2d7806e1f88.slice/crio-99eb4208cc89f9dd4fbaccf8d7b3e033dfabfa5d823e40a7b2dcd502eeb3d65e WatchSource:0}: Error finding container 99eb4208cc89f9dd4fbaccf8d7b3e033dfabfa5d823e40a7b2dcd502eeb3d65e: Status 404 returned error can't find the container with id 99eb4208cc89f9dd4fbaccf8d7b3e033dfabfa5d823e40a7b2dcd502eeb3d65e Oct 30 22:12:00.145146 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:12:00.145119 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:12:00.727688 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:00.727598 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e6dcc0-wjsj7" event={"ID":"b3d801e9-1bb2-4dc8-a8e0-c2d7806e1f88","Type":"ContainerStarted","Data":"bfd635cfd7766eb7c7948860f6de1d43f2f15304668811517899ec34cf52610a"} Oct 30 22:12:00.727688 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:00.727657 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e6dcc0-wjsj7" event={"ID":"b3d801e9-1bb2-4dc8-a8e0-c2d7806e1f88","Type":"ContainerStarted","Data":"99eb4208cc89f9dd4fbaccf8d7b3e033dfabfa5d823e40a7b2dcd502eeb3d65e"} Oct 30 22:12:00.866399 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:00.866338 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e6dcc0-wjsj7" podStartSLOduration=1.86632207 podStartE2EDuration="1.86632207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:00.864615229 +0000 UTC m=+2904.607136165" watchObservedRunningTime="2026-10-30 22:12:00.86632207 +0000 UTC m=+2904.608843005" Oct 30 22:12:01.734851 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:01.734811 5748 generic.go:356] "Generic (PLEG): container finished" podID="b3d801e9-1bb2-4dc8-a8e0-c2d7806e1f88" containerID="bfd635cfd7766eb7c7948860f6de1d43f2f15304668811517899ec34cf52610a" exitCode=0 Oct 30 22:12:01.735247 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:01.734889 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e6dcc0-wjsj7" event={"ID":"b3d801e9-1bb2-4dc8-a8e0-c2d7806e1f88","Type":"ContainerDied","Data":"bfd635cfd7766eb7c7948860f6de1d43f2f15304668811517899ec34cf52610a"} Oct 30 22:12:02.878650 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:02.878626 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e6dcc0-wjsj7" Oct 30 22:12:03.015987 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.015921 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3d801e9-1bb2-4dc8-a8e0-c2d7806e1f88-kube-api-access-mt465\" (UniqueName: \"kubernetes.io/projected/b3d801e9-1bb2-4dc8-a8e0-c2d7806e1f88-kube-api-access-mt465\") pod \"b3d801e9-1bb2-4dc8-a8e0-c2d7806e1f88\" (UID: \"b3d801e9-1bb2-4dc8-a8e0-c2d7806e1f88\") " Oct 30 22:12:03.016124 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.015989 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3d801e9-1bb2-4dc8-a8e0-c2d7806e1f88-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3d801e9-1bb2-4dc8-a8e0-c2d7806e1f88-maas-api-xks-ca\") pod \"b3d801e9-1bb2-4dc8-a8e0-c2d7806e1f88\" (UID: \"b3d801e9-1bb2-4dc8-a8e0-c2d7806e1f88\") " Oct 30 22:12:03.016124 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.016056 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3d801e9-1bb2-4dc8-a8e0-c2d7806e1f88-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3d801e9-1bb2-4dc8-a8e0-c2d7806e1f88-maas-api-openshift-ca\") pod \"b3d801e9-1bb2-4dc8-a8e0-c2d7806e1f88\" (UID: \"b3d801e9-1bb2-4dc8-a8e0-c2d7806e1f88\") " Oct 30 22:12:03.016418 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.016394 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3d801e9-1bb2-4dc8-a8e0-c2d7806e1f88-maas-api-openshift-ca" pod "b3d801e9-1bb2-4dc8-a8e0-c2d7806e1f88" (UID: "b3d801e9-1bb2-4dc8-a8e0-c2d7806e1f88"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:12:03.017921 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.017871 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3d801e9-1bb2-4dc8-a8e0-c2d7806e1f88-kube-api-access-mt465" pod "b3d801e9-1bb2-4dc8-a8e0-c2d7806e1f88" (UID: "b3d801e9-1bb2-4dc8-a8e0-c2d7806e1f88"). InnerVolumeSpecName "kube-api-access-mt465". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:12:03.018048 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.017962 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3d801e9-1bb2-4dc8-a8e0-c2d7806e1f88-maas-api-xks-ca" pod "b3d801e9-1bb2-4dc8-a8e0-c2d7806e1f88" (UID: "b3d801e9-1bb2-4dc8-a8e0-c2d7806e1f88"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:12:03.101433 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.101384 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-c7d477/distinct-w5-c7d477-kserve-554d7f974d-6zw6s" podUID="09d39cc0-b68d-48fd-918c-7d2518f384aa" containerName="main" containerID="cri-o://5e7bd4ac6f184e2668da1bea983d9cb15db3fb7acc2ff5f204651708c2b53383" gracePeriod=45 Oct 30 22:12:03.117119 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.117095 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mt465\" (UniqueName: \"kubernetes.io/projected/b3d801e9-1bb2-4dc8-a8e0-c2d7806e1f88-kube-api-access-mt465\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:03.117210 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.117125 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3d801e9-1bb2-4dc8-a8e0-c2d7806e1f88-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:03.117210 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.117140 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3d801e9-1bb2-4dc8-a8e0-c2d7806e1f88-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:03.121167 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.121139 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h" podUID="0a0b0425-2959-4690-9847-38852e167230" containerName="main" containerID="cri-o://2cb1be98fa914075f3145977294d5c030762062ddf458e3c7cf24fa93cbbbcc9" gracePeriod=45 Oct 30 22:12:03.151556 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.151529 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h" podUID="0a0b0425-2959-4690-9847-38852e167230" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": dial tcp 10.134.0.43:8000: connect: connection refused" Oct 30 22:12:03.360187 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.360156 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c7d477/distinct-w5-c7d477-kserve-554d7f974d-6zw6s" Oct 30 22:12:03.371016 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.370995 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h" Oct 30 22:12:03.420525 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.420496 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09d39cc0-b68d-48fd-918c-7d2518f384aa-home\" (UniqueName: \"kubernetes.io/empty-dir/09d39cc0-b68d-48fd-918c-7d2518f384aa-home\") pod \"09d39cc0-b68d-48fd-918c-7d2518f384aa\" (UID: \"09d39cc0-b68d-48fd-918c-7d2518f384aa\") " Oct 30 22:12:03.420673 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.420566 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a0b0425-2959-4690-9847-38852e167230-home\" (UniqueName: \"kubernetes.io/empty-dir/0a0b0425-2959-4690-9847-38852e167230-home\") pod \"0a0b0425-2959-4690-9847-38852e167230\" (UID: \"0a0b0425-2959-4690-9847-38852e167230\") " Oct 30 22:12:03.420673 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.420624 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09d39cc0-b68d-48fd-918c-7d2518f384aa-kube-api-access-2mkjj\" (UniqueName: \"kubernetes.io/projected/09d39cc0-b68d-48fd-918c-7d2518f384aa-kube-api-access-2mkjj\") pod \"09d39cc0-b68d-48fd-918c-7d2518f384aa\" (UID: \"09d39cc0-b68d-48fd-918c-7d2518f384aa\") " Oct 30 22:12:03.420673 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.420656 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09d39cc0-b68d-48fd-918c-7d2518f384aa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09d39cc0-b68d-48fd-918c-7d2518f384aa-tmp-dir\") pod \"09d39cc0-b68d-48fd-918c-7d2518f384aa\" (UID: \"09d39cc0-b68d-48fd-918c-7d2518f384aa\") " Oct 30 22:12:03.420855 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.420714 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a0b0425-2959-4690-9847-38852e167230-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a0b0425-2959-4690-9847-38852e167230-tmp-dir\") pod \"0a0b0425-2959-4690-9847-38852e167230\" (UID: \"0a0b0425-2959-4690-9847-38852e167230\") " Oct 30 22:12:03.420855 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.420736 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09d39cc0-b68d-48fd-918c-7d2518f384aa-home" pod "09d39cc0-b68d-48fd-918c-7d2518f384aa" (UID: "09d39cc0-b68d-48fd-918c-7d2518f384aa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:12:03.420855 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.420826 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a0b0425-2959-4690-9847-38852e167230-home" pod "0a0b0425-2959-4690-9847-38852e167230" (UID: "0a0b0425-2959-4690-9847-38852e167230"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:12:03.420855 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.420834 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a0b0425-2959-4690-9847-38852e167230-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a0b0425-2959-4690-9847-38852e167230-dshm\") pod \"0a0b0425-2959-4690-9847-38852e167230\" (UID: \"0a0b0425-2959-4690-9847-38852e167230\") " Oct 30 22:12:03.421094 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.420875 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a0b0425-2959-4690-9847-38852e167230-tls-certs\" (UniqueName: \"kubernetes.io/secret/0a0b0425-2959-4690-9847-38852e167230-tls-certs\") pod \"0a0b0425-2959-4690-9847-38852e167230\" (UID: \"0a0b0425-2959-4690-9847-38852e167230\") " Oct 30 22:12:03.421094 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.420931 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09d39cc0-b68d-48fd-918c-7d2518f384aa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09d39cc0-b68d-48fd-918c-7d2518f384aa-model-cache\") pod \"09d39cc0-b68d-48fd-918c-7d2518f384aa\" (UID: \"09d39cc0-b68d-48fd-918c-7d2518f384aa\") " Oct 30 22:12:03.421094 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.420967 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09d39cc0-b68d-48fd-918c-7d2518f384aa-tls-certs\" (UniqueName: \"kubernetes.io/secret/09d39cc0-b68d-48fd-918c-7d2518f384aa-tls-certs\") pod \"09d39cc0-b68d-48fd-918c-7d2518f384aa\" (UID: \"09d39cc0-b68d-48fd-918c-7d2518f384aa\") " Oct 30 22:12:03.421094 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.420967 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09d39cc0-b68d-48fd-918c-7d2518f384aa-tmp-dir" pod "09d39cc0-b68d-48fd-918c-7d2518f384aa" (UID: "09d39cc0-b68d-48fd-918c-7d2518f384aa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:12:03.421094 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.420976 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a0b0425-2959-4690-9847-38852e167230-tmp-dir" pod "0a0b0425-2959-4690-9847-38852e167230" (UID: "0a0b0425-2959-4690-9847-38852e167230"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:12:03.421094 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.420996 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a0b0425-2959-4690-9847-38852e167230-kube-api-access-cmnrq\" (UniqueName: \"kubernetes.io/projected/0a0b0425-2959-4690-9847-38852e167230-kube-api-access-cmnrq\") pod \"0a0b0425-2959-4690-9847-38852e167230\" (UID: \"0a0b0425-2959-4690-9847-38852e167230\") " Oct 30 22:12:03.421094 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.421095 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09d39cc0-b68d-48fd-918c-7d2518f384aa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/09d39cc0-b68d-48fd-918c-7d2518f384aa-dshm\") pod \"09d39cc0-b68d-48fd-918c-7d2518f384aa\" (UID: \"09d39cc0-b68d-48fd-918c-7d2518f384aa\") " Oct 30 22:12:03.421621 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.421145 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a0b0425-2959-4690-9847-38852e167230-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a0b0425-2959-4690-9847-38852e167230-model-cache\") pod \"0a0b0425-2959-4690-9847-38852e167230\" (UID: \"0a0b0425-2959-4690-9847-38852e167230\") " Oct 30 22:12:03.421621 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.421159 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09d39cc0-b68d-48fd-918c-7d2518f384aa-model-cache" pod "09d39cc0-b68d-48fd-918c-7d2518f384aa" (UID: "09d39cc0-b68d-48fd-918c-7d2518f384aa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:12:03.421621 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.421437 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a0b0425-2959-4690-9847-38852e167230-model-cache" pod "0a0b0425-2959-4690-9847-38852e167230" (UID: "0a0b0425-2959-4690-9847-38852e167230"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:12:03.421621 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.421551 5748 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a0b0425-2959-4690-9847-38852e167230-home\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:03.421621 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.421592 5748 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09d39cc0-b68d-48fd-918c-7d2518f384aa-tmp-dir\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:03.421621 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.421609 5748 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a0b0425-2959-4690-9847-38852e167230-tmp-dir\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:03.421621 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.421622 5748 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09d39cc0-b68d-48fd-918c-7d2518f384aa-model-cache\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:03.422047 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.421638 5748 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a0b0425-2959-4690-9847-38852e167230-model-cache\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:03.422047 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.421652 5748 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09d39cc0-b68d-48fd-918c-7d2518f384aa-home\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:03.423277 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.423256 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09d39cc0-b68d-48fd-918c-7d2518f384aa-tls-certs" pod "09d39cc0-b68d-48fd-918c-7d2518f384aa" (UID: "09d39cc0-b68d-48fd-918c-7d2518f384aa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:12:03.423406 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.423288 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09d39cc0-b68d-48fd-918c-7d2518f384aa-kube-api-access-2mkjj" pod "09d39cc0-b68d-48fd-918c-7d2518f384aa" (UID: "09d39cc0-b68d-48fd-918c-7d2518f384aa"). InnerVolumeSpecName "kube-api-access-2mkjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:12:03.423406 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.423365 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a0b0425-2959-4690-9847-38852e167230-dshm" pod "0a0b0425-2959-4690-9847-38852e167230" (UID: "0a0b0425-2959-4690-9847-38852e167230"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:12:03.423529 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.423483 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a0b0425-2959-4690-9847-38852e167230-tls-certs" pod "0a0b0425-2959-4690-9847-38852e167230" (UID: "0a0b0425-2959-4690-9847-38852e167230"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:12:03.423633 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.423608 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09d39cc0-b68d-48fd-918c-7d2518f384aa-dshm" pod "09d39cc0-b68d-48fd-918c-7d2518f384aa" (UID: "09d39cc0-b68d-48fd-918c-7d2518f384aa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:12:03.423837 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.423816 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a0b0425-2959-4690-9847-38852e167230-kube-api-access-cmnrq" pod "0a0b0425-2959-4690-9847-38852e167230" (UID: "0a0b0425-2959-4690-9847-38852e167230"). InnerVolumeSpecName "kube-api-access-cmnrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:12:03.522452 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.522422 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mkjj\" (UniqueName: \"kubernetes.io/projected/09d39cc0-b68d-48fd-918c-7d2518f384aa-kube-api-access-2mkjj\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:03.522452 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.522452 5748 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a0b0425-2959-4690-9847-38852e167230-dshm\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:03.522764 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.522503 5748 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a0b0425-2959-4690-9847-38852e167230-tls-certs\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:03.522764 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.522518 5748 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09d39cc0-b68d-48fd-918c-7d2518f384aa-tls-certs\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:03.522764 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.522560 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmnrq\" (UniqueName: \"kubernetes.io/projected/0a0b0425-2959-4690-9847-38852e167230-kube-api-access-cmnrq\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:03.522764 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.522576 5748 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09d39cc0-b68d-48fd-918c-7d2518f384aa-dshm\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:03.747969 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.747932 5748 generic.go:356] "Generic (PLEG): container finished" podID="0a0b0425-2959-4690-9847-38852e167230" containerID="2cb1be98fa914075f3145977294d5c030762062ddf458e3c7cf24fa93cbbbcc9" exitCode=0 Oct 30 22:12:03.748157 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.747974 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h" Oct 30 22:12:03.748157 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.748006 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h" event={"ID":"0a0b0425-2959-4690-9847-38852e167230","Type":"ContainerDied","Data":"2cb1be98fa914075f3145977294d5c030762062ddf458e3c7cf24fa93cbbbcc9"} Oct 30 22:12:03.748157 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.748050 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h" event={"ID":"0a0b0425-2959-4690-9847-38852e167230","Type":"ContainerDied","Data":"819a71e6830606f37af1fd1adcf2903b2f503912836e3f808e5234f9ce8f242c"} Oct 30 22:12:03.748157 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.748066 5748 scope.go:122] "RemoveContainer" containerID="2cb1be98fa914075f3145977294d5c030762062ddf458e3c7cf24fa93cbbbcc9" Oct 30 22:12:03.749544 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.749518 5748 generic.go:356] "Generic (PLEG): container finished" podID="09d39cc0-b68d-48fd-918c-7d2518f384aa" containerID="5e7bd4ac6f184e2668da1bea983d9cb15db3fb7acc2ff5f204651708c2b53383" exitCode=0 Oct 30 22:12:03.749663 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.749562 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c7d477/distinct-w5-c7d477-kserve-554d7f974d-6zw6s" Oct 30 22:12:03.749663 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.749615 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c7d477/distinct-w5-c7d477-kserve-554d7f974d-6zw6s" event={"ID":"09d39cc0-b68d-48fd-918c-7d2518f384aa","Type":"ContainerDied","Data":"5e7bd4ac6f184e2668da1bea983d9cb15db3fb7acc2ff5f204651708c2b53383"} Oct 30 22:12:03.749663 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.749646 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c7d477/distinct-w5-c7d477-kserve-554d7f974d-6zw6s" event={"ID":"09d39cc0-b68d-48fd-918c-7d2518f384aa","Type":"ContainerDied","Data":"310e655363aecc4d77163999c17ad6443cdd05aa943e9555888f9c70fac056b7"} Oct 30 22:12:03.751542 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.751525 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e6dcc0-wjsj7" Oct 30 22:12:03.751782 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.751527 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e6dcc0-wjsj7" event={"ID":"b3d801e9-1bb2-4dc8-a8e0-c2d7806e1f88","Type":"ContainerDied","Data":"99eb4208cc89f9dd4fbaccf8d7b3e033dfabfa5d823e40a7b2dcd502eeb3d65e"} Oct 30 22:12:03.751919 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.751807 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99eb4208cc89f9dd4fbaccf8d7b3e033dfabfa5d823e40a7b2dcd502eeb3d65e" Oct 30 22:12:03.771038 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.768148 5748 scope.go:122] "RemoveContainer" containerID="2cb1be98fa914075f3145977294d5c030762062ddf458e3c7cf24fa93cbbbcc9" Oct 30 22:12:03.771261 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:12:03.771236 5748 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cb1be98fa914075f3145977294d5c030762062ddf458e3c7cf24fa93cbbbcc9\": container with ID starting with 2cb1be98fa914075f3145977294d5c030762062ddf458e3c7cf24fa93cbbbcc9 not found: ID does not exist" containerID="2cb1be98fa914075f3145977294d5c030762062ddf458e3c7cf24fa93cbbbcc9" Oct 30 22:12:03.771357 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.771271 5748 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cb1be98fa914075f3145977294d5c030762062ddf458e3c7cf24fa93cbbbcc9"} err="failed to get container status \"2cb1be98fa914075f3145977294d5c030762062ddf458e3c7cf24fa93cbbbcc9\": rpc error: code = NotFound desc = could not find container \"2cb1be98fa914075f3145977294d5c030762062ddf458e3c7cf24fa93cbbbcc9\": container with ID starting with 2cb1be98fa914075f3145977294d5c030762062ddf458e3c7cf24fa93cbbbcc9 not found: ID does not exist" Oct 30 22:12:03.771357 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.771294 5748 scope.go:122] "RemoveContainer" containerID="5e7bd4ac6f184e2668da1bea983d9cb15db3fb7acc2ff5f204651708c2b53383" Oct 30 22:12:03.780868 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.780844 5748 scope.go:122] "RemoveContainer" containerID="5e7bd4ac6f184e2668da1bea983d9cb15db3fb7acc2ff5f204651708c2b53383" Oct 30 22:12:03.781150 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:12:03.781131 5748 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e7bd4ac6f184e2668da1bea983d9cb15db3fb7acc2ff5f204651708c2b53383\": container with ID starting with 5e7bd4ac6f184e2668da1bea983d9cb15db3fb7acc2ff5f204651708c2b53383 not found: ID does not exist" containerID="5e7bd4ac6f184e2668da1bea983d9cb15db3fb7acc2ff5f204651708c2b53383" Oct 30 22:12:03.781215 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.781155 5748 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e7bd4ac6f184e2668da1bea983d9cb15db3fb7acc2ff5f204651708c2b53383"} err="failed to get container status \"5e7bd4ac6f184e2668da1bea983d9cb15db3fb7acc2ff5f204651708c2b53383\": rpc error: code = NotFound desc = could not find container \"5e7bd4ac6f184e2668da1bea983d9cb15db3fb7acc2ff5f204651708c2b53383\": container with ID starting with 5e7bd4ac6f184e2668da1bea983d9cb15db3fb7acc2ff5f204651708c2b53383 not found: ID does not exist" Oct 30 22:12:03.936160 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.936126 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-c7d477/distinct-w5-c7d477-kserve-554d7f974d-6zw6s"] Oct 30 22:12:03.954665 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:03.954633 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-c7d477/distinct-w5-c7d477-kserve-554d7f974d-6zw6s"] Oct 30 22:12:04.137966 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:04.137938 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h"] Oct 30 22:12:04.143468 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:04.143441 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-c7d477/simulator-w5-c7d477-kserve-9f759d5bd-59r8h"] Oct 30 22:12:04.416352 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:04.416275 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09d39cc0-b68d-48fd-918c-7d2518f384aa" path="/var/lib/kubelet/pods/09d39cc0-b68d-48fd-918c-7d2518f384aa/volumes" Oct 30 22:12:04.416704 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:04.416691 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a0b0425-2959-4690-9847-38852e167230" path="/var/lib/kubelet/pods/0a0b0425-2959-4690-9847-38852e167230/volumes" Oct 30 22:12:23.542885 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:23.542849 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c7d477-5nl7t"] Oct 30 22:12:23.593117 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:23.593082 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c7d477-5nl7t"] Oct 30 22:12:24.424660 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:24.424623 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ba5816c-29e8-4d05-b67b-1674503fb759" path="/var/lib/kubelet/pods/4ba5816c-29e8-4d05-b67b-1674503fb759/volumes" Oct 30 22:12:30.144745 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:12:30.144706 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:12:30.250273 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:30.250238 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e6dcc0-wjsj7"] Oct 30 22:12:30.300281 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:30.300251 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e6dcc0-wjsj7"] Oct 30 22:12:30.419742 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:30.419655 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3d801e9-1bb2-4dc8-a8e0-c2d7806e1f88" path="/var/lib/kubelet/pods/b3d801e9-1bb2-4dc8-a8e0-c2d7806e1f88/volumes" Oct 30 22:12:38.440359 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:38.440324 5748 scope.go:122] "RemoveContainer" containerID="caef26ad7fb09979be58ff2182e48c7b9170d24e5fc4b8f531db174df8790935" Oct 30 22:12:38.449280 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:38.449260 5748 scope.go:122] "RemoveContainer" containerID="e8d2fd0cb71df71c55152fd7a66c73eb5f4cc40e8dc5f1b96ab7d7b2361e1292" Oct 30 22:12:38.457477 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:38.457451 5748 scope.go:122] "RemoveContainer" containerID="93fb0d7a3eb48102b4bc96a8b599df04eae9664c37f3ec7dc9dbb2a89a48f52f" Oct 30 22:12:48.415872 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:48.415836 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19bc92-fjkmf"] Oct 30 22:12:48.416445 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:48.416422 5748 cpu_manager.go:414] "containerMap: removing container" podUID="b3d801e9-1bb2-4dc8-a8e0-c2d7806e1f88" containerName="revoke-keys" Oct 30 22:12:48.416548 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:48.416447 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3d801e9-1bb2-4dc8-a8e0-c2d7806e1f88" containerName="revoke-keys" Oct 30 22:12:48.416548 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:48.416492 5748 cpu_manager.go:414] "containerMap: removing container" podUID="09d39cc0-b68d-48fd-918c-7d2518f384aa" containerName="main" Oct 30 22:12:48.416548 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:48.416500 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09d39cc0-b68d-48fd-918c-7d2518f384aa" containerName="main" Oct 30 22:12:48.416548 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:48.416527 5748 cpu_manager.go:414] "containerMap: removing container" podUID="0a0b0425-2959-4690-9847-38852e167230" containerName="main" Oct 30 22:12:48.416548 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:48.416535 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a0b0425-2959-4690-9847-38852e167230" containerName="main" Oct 30 22:12:48.416813 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:48.416643 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a0b0425-2959-4690-9847-38852e167230" containerName="main" Oct 30 22:12:48.416813 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:48.416661 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3d801e9-1bb2-4dc8-a8e0-c2d7806e1f88" containerName="revoke-keys" Oct 30 22:12:48.416813 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:48.416676 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="09d39cc0-b68d-48fd-918c-7d2518f384aa" containerName="main" Oct 30 22:12:48.421344 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:48.421323 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19bc92-fjkmf" Oct 30 22:12:48.484395 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:48.484358 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19bc92-fjkmf"] Oct 30 22:12:48.531680 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:48.531647 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:12:48.531680 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:48.531670 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:12:48.531680 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:48.531676 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:12:48.531928 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:48.531715 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 22:12:48.588541 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:48.588513 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c8b3a66-c6ec-4ff2-a10c-0221c729af89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-19bc92-fjkmf\" (UID: \"3c8b3a66-c6ec-4ff2-a10c-0221c729af89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19bc92-fjkmf" Oct 30 22:12:48.588694 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:48.588678 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqqwb\" (UniqueName: \"kubernetes.io/projected/3c8b3a66-c6ec-4ff2-a10c-0221c729af89-kube-api-access-gqqwb\") pod \"maas-api-revoke-keys-e2e-inf-b-19bc92-fjkmf\" (UID: \"3c8b3a66-c6ec-4ff2-a10c-0221c729af89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19bc92-fjkmf" Oct 30 22:12:48.588743 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:48.588712 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c8b3a66-c6ec-4ff2-a10c-0221c729af89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-19bc92-fjkmf\" (UID: \"3c8b3a66-c6ec-4ff2-a10c-0221c729af89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19bc92-fjkmf" Oct 30 22:12:48.689756 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:48.689695 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqqwb\" (UniqueName: \"kubernetes.io/projected/3c8b3a66-c6ec-4ff2-a10c-0221c729af89-kube-api-access-gqqwb\") pod \"maas-api-revoke-keys-e2e-inf-b-19bc92-fjkmf\" (UID: \"3c8b3a66-c6ec-4ff2-a10c-0221c729af89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19bc92-fjkmf" Oct 30 22:12:48.689756 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:48.689731 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c8b3a66-c6ec-4ff2-a10c-0221c729af89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-19bc92-fjkmf\" (UID: \"3c8b3a66-c6ec-4ff2-a10c-0221c729af89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19bc92-fjkmf" Oct 30 22:12:48.689756 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:48.689748 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c8b3a66-c6ec-4ff2-a10c-0221c729af89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-19bc92-fjkmf\" (UID: \"3c8b3a66-c6ec-4ff2-a10c-0221c729af89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19bc92-fjkmf" Oct 30 22:12:48.690498 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:48.690477 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c8b3a66-c6ec-4ff2-a10c-0221c729af89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-19bc92-fjkmf\" (UID: \"3c8b3a66-c6ec-4ff2-a10c-0221c729af89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19bc92-fjkmf" Oct 30 22:12:48.692043 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:48.692023 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c8b3a66-c6ec-4ff2-a10c-0221c729af89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-19bc92-fjkmf\" (UID: \"3c8b3a66-c6ec-4ff2-a10c-0221c729af89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19bc92-fjkmf" Oct 30 22:12:48.752569 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:48.752546 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqqwb\" (UniqueName: \"kubernetes.io/projected/3c8b3a66-c6ec-4ff2-a10c-0221c729af89-kube-api-access-gqqwb\") pod \"maas-api-revoke-keys-e2e-inf-b-19bc92-fjkmf\" (UID: \"3c8b3a66-c6ec-4ff2-a10c-0221c729af89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19bc92-fjkmf" Oct 30 22:12:49.034209 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:49.034130 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19bc92-fjkmf" Oct 30 22:12:49.433499 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:49.433369 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19bc92-fjkmf"] Oct 30 22:12:49.436684 ip-10-0-143-61 kubenswrapper[5748]: W1030 22:12:49.436654 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c8b3a66_c6ec_4ff2_a10c_0221c729af89.slice/crio-2afad391d9f5e5ede2eb23cd63b06575088ed353856fa15aa0badf05e888db0a WatchSource:0}: Error finding container 2afad391d9f5e5ede2eb23cd63b06575088ed353856fa15aa0badf05e888db0a: Status 404 returned error can't find the container with id 2afad391d9f5e5ede2eb23cd63b06575088ed353856fa15aa0badf05e888db0a Oct 30 22:12:50.031862 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:50.031774 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19bc92-fjkmf" event={"ID":"3c8b3a66-c6ec-4ff2-a10c-0221c729af89","Type":"ContainerStarted","Data":"bbb54838fdde62a75e2bf442b14182076781a635ec8eaa2a73ae1be83deb4144"} Oct 30 22:12:50.031862 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:50.031829 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19bc92-fjkmf" event={"ID":"3c8b3a66-c6ec-4ff2-a10c-0221c729af89","Type":"ContainerStarted","Data":"2afad391d9f5e5ede2eb23cd63b06575088ed353856fa15aa0badf05e888db0a"} Oct 30 22:12:51.039570 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:51.039534 5748 generic.go:356] "Generic (PLEG): container finished" podID="3c8b3a66-c6ec-4ff2-a10c-0221c729af89" containerID="bbb54838fdde62a75e2bf442b14182076781a635ec8eaa2a73ae1be83deb4144" exitCode=0 Oct 30 22:12:51.040009 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:51.039620 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19bc92-fjkmf" event={"ID":"3c8b3a66-c6ec-4ff2-a10c-0221c729af89","Type":"ContainerDied","Data":"bbb54838fdde62a75e2bf442b14182076781a635ec8eaa2a73ae1be83deb4144"} Oct 30 22:12:52.175652 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:52.175626 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19bc92-fjkmf" Oct 30 22:12:52.222847 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:52.222820 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c8b3a66-c6ec-4ff2-a10c-0221c729af89-kube-api-access-gqqwb\" (UniqueName: \"kubernetes.io/projected/3c8b3a66-c6ec-4ff2-a10c-0221c729af89-kube-api-access-gqqwb\") pod \"3c8b3a66-c6ec-4ff2-a10c-0221c729af89\" (UID: \"3c8b3a66-c6ec-4ff2-a10c-0221c729af89\") " Oct 30 22:12:52.223015 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:52.222920 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c8b3a66-c6ec-4ff2-a10c-0221c729af89-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c8b3a66-c6ec-4ff2-a10c-0221c729af89-maas-api-openshift-ca\") pod \"3c8b3a66-c6ec-4ff2-a10c-0221c729af89\" (UID: \"3c8b3a66-c6ec-4ff2-a10c-0221c729af89\") " Oct 30 22:12:52.223015 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:52.223000 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c8b3a66-c6ec-4ff2-a10c-0221c729af89-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c8b3a66-c6ec-4ff2-a10c-0221c729af89-maas-api-xks-ca\") pod \"3c8b3a66-c6ec-4ff2-a10c-0221c729af89\" (UID: \"3c8b3a66-c6ec-4ff2-a10c-0221c729af89\") " Oct 30 22:12:52.223254 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:52.223224 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c8b3a66-c6ec-4ff2-a10c-0221c729af89-maas-api-openshift-ca" pod "3c8b3a66-c6ec-4ff2-a10c-0221c729af89" (UID: "3c8b3a66-c6ec-4ff2-a10c-0221c729af89"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:12:52.224964 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:52.224944 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c8b3a66-c6ec-4ff2-a10c-0221c729af89-maas-api-xks-ca" pod "3c8b3a66-c6ec-4ff2-a10c-0221c729af89" (UID: "3c8b3a66-c6ec-4ff2-a10c-0221c729af89"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:12:52.225059 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:52.224945 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c8b3a66-c6ec-4ff2-a10c-0221c729af89-kube-api-access-gqqwb" pod "3c8b3a66-c6ec-4ff2-a10c-0221c729af89" (UID: "3c8b3a66-c6ec-4ff2-a10c-0221c729af89"). InnerVolumeSpecName "kube-api-access-gqqwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:12:52.324030 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:52.323957 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqqwb\" (UniqueName: \"kubernetes.io/projected/3c8b3a66-c6ec-4ff2-a10c-0221c729af89-kube-api-access-gqqwb\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:52.324030 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:52.323991 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c8b3a66-c6ec-4ff2-a10c-0221c729af89-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:52.324030 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:52.324007 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c8b3a66-c6ec-4ff2-a10c-0221c729af89-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:53.051739 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:53.051708 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19bc92-fjkmf" Oct 30 22:12:53.051963 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:53.051748 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19bc92-fjkmf" event={"ID":"3c8b3a66-c6ec-4ff2-a10c-0221c729af89","Type":"ContainerDied","Data":"2afad391d9f5e5ede2eb23cd63b06575088ed353856fa15aa0badf05e888db0a"} Oct 30 22:12:53.051963 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:12:53.051795 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2afad391d9f5e5ede2eb23cd63b06575088ed353856fa15aa0badf05e888db0a" Oct 30 22:13:00.148149 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:13:00.148110 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:13:05.158955 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:05.158917 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19bc92-fjkmf"] Oct 30 22:13:05.208088 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:05.208059 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19bc92-fjkmf"] Oct 30 22:13:06.416661 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:06.416624 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c8b3a66-c6ec-4ff2-a10c-0221c729af89" path="/var/lib/kubelet/pods/3c8b3a66-c6ec-4ff2-a10c-0221c729af89/volumes" Oct 30 22:13:12.112341 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:12.112257 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v"] Oct 30 22:13:12.112721 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:12.112477 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v" podUID="024f007a-0f46-4093-8554-93a335039f10" containerName="istio-proxy" containerID="cri-o://a07053a9af92e329f37abf16ea55d304cf51778fe8301ed34cedca2613fe3319" gracePeriod=30 Oct 30 22:13:13.138753 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:13.138715 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-69e4b2-62tk5"] Oct 30 22:13:13.139173 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:13.139158 5748 cpu_manager.go:414] "containerMap: removing container" podUID="3c8b3a66-c6ec-4ff2-a10c-0221c729af89" containerName="revoke-keys" Oct 30 22:13:13.139173 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:13.139172 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c8b3a66-c6ec-4ff2-a10c-0221c729af89" containerName="revoke-keys" Oct 30 22:13:13.139255 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:13.139245 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c8b3a66-c6ec-4ff2-a10c-0221c729af89" containerName="revoke-keys" Oct 30 22:13:13.142513 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:13.142497 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-69e4b2-62tk5" Oct 30 22:13:13.198617 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:13.198580 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:13:13.198795 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:13.198656 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:13:13.198795 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:13.198657 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:13:13.198795 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:13.198723 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 22:13:13.205126 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:13.205104 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-69e4b2-62tk5"] Oct 30 22:13:13.225726 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:13.225697 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1370cc1-be39-4620-93c5-54d69fb6ef3b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-69e4b2-62tk5\" (UID: \"e1370cc1-be39-4620-93c5-54d69fb6ef3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-69e4b2-62tk5" Oct 30 22:13:13.225859 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:13.225744 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkrmj\" (UniqueName: \"kubernetes.io/projected/e1370cc1-be39-4620-93c5-54d69fb6ef3b-kube-api-access-bkrmj\") pod \"maas-api-revoke-keys-e2e-worker-w6-69e4b2-62tk5\" (UID: \"e1370cc1-be39-4620-93c5-54d69fb6ef3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-69e4b2-62tk5" Oct 30 22:13:13.225975 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:13.225950 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1370cc1-be39-4620-93c5-54d69fb6ef3b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-69e4b2-62tk5\" (UID: \"e1370cc1-be39-4620-93c5-54d69fb6ef3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-69e4b2-62tk5" Oct 30 22:13:13.326952 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:13.326914 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1370cc1-be39-4620-93c5-54d69fb6ef3b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-69e4b2-62tk5\" (UID: \"e1370cc1-be39-4620-93c5-54d69fb6ef3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-69e4b2-62tk5" Oct 30 22:13:13.327113 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:13.326967 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1370cc1-be39-4620-93c5-54d69fb6ef3b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-69e4b2-62tk5\" (UID: \"e1370cc1-be39-4620-93c5-54d69fb6ef3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-69e4b2-62tk5" Oct 30 22:13:13.327113 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:13.327003 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkrmj\" (UniqueName: \"kubernetes.io/projected/e1370cc1-be39-4620-93c5-54d69fb6ef3b-kube-api-access-bkrmj\") pod \"maas-api-revoke-keys-e2e-worker-w6-69e4b2-62tk5\" (UID: \"e1370cc1-be39-4620-93c5-54d69fb6ef3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-69e4b2-62tk5" Oct 30 22:13:13.327616 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:13.327590 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1370cc1-be39-4620-93c5-54d69fb6ef3b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-69e4b2-62tk5\" (UID: \"e1370cc1-be39-4620-93c5-54d69fb6ef3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-69e4b2-62tk5" Oct 30 22:13:13.329169 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:13.329150 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1370cc1-be39-4620-93c5-54d69fb6ef3b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-69e4b2-62tk5\" (UID: \"e1370cc1-be39-4620-93c5-54d69fb6ef3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-69e4b2-62tk5" Oct 30 22:13:13.388884 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:13.388814 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkrmj\" (UniqueName: \"kubernetes.io/projected/e1370cc1-be39-4620-93c5-54d69fb6ef3b-kube-api-access-bkrmj\") pod \"maas-api-revoke-keys-e2e-worker-w6-69e4b2-62tk5\" (UID: \"e1370cc1-be39-4620-93c5-54d69fb6ef3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-69e4b2-62tk5" Oct 30 22:13:13.455825 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:13.455800 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-69e4b2-62tk5" Oct 30 22:13:13.681672 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:13.681594 5748 patch_prober.go:29] interesting pod/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:13:13.681672 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:13.681644 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v" podUID="024f007a-0f46-4093-8554-93a335039f10" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:13:13.829705 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:13.829673 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-69e4b2-62tk5"] Oct 30 22:13:13.831589 ip-10-0-143-61 kubenswrapper[5748]: W1030 22:13:13.831546 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1370cc1_be39_4620_93c5_54d69fb6ef3b.slice/crio-fe3a13a56832aa2f396fcaf82bba09e204af67cc2a624cd4545718ee15ca6c49 WatchSource:0}: Error finding container fe3a13a56832aa2f396fcaf82bba09e204af67cc2a624cd4545718ee15ca6c49: Status 404 returned error can't find the container with id fe3a13a56832aa2f396fcaf82bba09e204af67cc2a624cd4545718ee15ca6c49 Oct 30 22:13:14.173810 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:14.173766 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-69e4b2-62tk5" event={"ID":"e1370cc1-be39-4620-93c5-54d69fb6ef3b","Type":"ContainerStarted","Data":"1457707154a239c34dacc2beb9c6edcbc33cc64681cb708d8be17c3eb91a10f5"} Oct 30 22:13:14.174305 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:14.173833 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-69e4b2-62tk5" event={"ID":"e1370cc1-be39-4620-93c5-54d69fb6ef3b","Type":"ContainerStarted","Data":"fe3a13a56832aa2f396fcaf82bba09e204af67cc2a624cd4545718ee15ca6c49"} Oct 30 22:13:14.293117 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:14.293023 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-69e4b2-62tk5" podStartSLOduration=1.293008422 podStartE2EDuration="1.293008422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:14.290687344 +0000 UTC m=+2978.033208279" watchObservedRunningTime="2026-10-30 22:13:14.293008422 +0000 UTC m=+2978.035529356" Oct 30 22:13:15.181514 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:15.181481 5748 generic.go:356] "Generic (PLEG): container finished" podID="e1370cc1-be39-4620-93c5-54d69fb6ef3b" containerID="1457707154a239c34dacc2beb9c6edcbc33cc64681cb708d8be17c3eb91a10f5" exitCode=0 Oct 30 22:13:15.181923 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:15.181571 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-69e4b2-62tk5" event={"ID":"e1370cc1-be39-4620-93c5-54d69fb6ef3b","Type":"ContainerDied","Data":"1457707154a239c34dacc2beb9c6edcbc33cc64681cb708d8be17c3eb91a10f5"} Oct 30 22:13:16.323476 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:16.323445 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-69e4b2-62tk5" Oct 30 22:13:16.459027 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:16.458952 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1370cc1-be39-4620-93c5-54d69fb6ef3b-kube-api-access-bkrmj\" (UniqueName: \"kubernetes.io/projected/e1370cc1-be39-4620-93c5-54d69fb6ef3b-kube-api-access-bkrmj\") pod \"e1370cc1-be39-4620-93c5-54d69fb6ef3b\" (UID: \"e1370cc1-be39-4620-93c5-54d69fb6ef3b\") " Oct 30 22:13:16.459195 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:16.459069 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1370cc1-be39-4620-93c5-54d69fb6ef3b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1370cc1-be39-4620-93c5-54d69fb6ef3b-maas-api-openshift-ca\") pod \"e1370cc1-be39-4620-93c5-54d69fb6ef3b\" (UID: \"e1370cc1-be39-4620-93c5-54d69fb6ef3b\") " Oct 30 22:13:16.459195 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:16.459101 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1370cc1-be39-4620-93c5-54d69fb6ef3b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1370cc1-be39-4620-93c5-54d69fb6ef3b-maas-api-xks-ca\") pod \"e1370cc1-be39-4620-93c5-54d69fb6ef3b\" (UID: \"e1370cc1-be39-4620-93c5-54d69fb6ef3b\") " Oct 30 22:13:16.459419 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:16.459392 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1370cc1-be39-4620-93c5-54d69fb6ef3b-maas-api-openshift-ca" pod "e1370cc1-be39-4620-93c5-54d69fb6ef3b" (UID: "e1370cc1-be39-4620-93c5-54d69fb6ef3b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:13:16.460955 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:16.460932 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1370cc1-be39-4620-93c5-54d69fb6ef3b-kube-api-access-bkrmj" pod "e1370cc1-be39-4620-93c5-54d69fb6ef3b" (UID: "e1370cc1-be39-4620-93c5-54d69fb6ef3b"). InnerVolumeSpecName "kube-api-access-bkrmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:16.461112 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:16.461092 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1370cc1-be39-4620-93c5-54d69fb6ef3b-maas-api-xks-ca" pod "e1370cc1-be39-4620-93c5-54d69fb6ef3b" (UID: "e1370cc1-be39-4620-93c5-54d69fb6ef3b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:13:16.560358 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:16.560331 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1370cc1-be39-4620-93c5-54d69fb6ef3b-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:16.560358 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:16.560359 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1370cc1-be39-4620-93c5-54d69fb6ef3b-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:16.560358 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:16.560369 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkrmj\" (UniqueName: \"kubernetes.io/projected/e1370cc1-be39-4620-93c5-54d69fb6ef3b-kube-api-access-bkrmj\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:17.194229 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:17.194194 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-69e4b2-62tk5" event={"ID":"e1370cc1-be39-4620-93c5-54d69fb6ef3b","Type":"ContainerDied","Data":"fe3a13a56832aa2f396fcaf82bba09e204af67cc2a624cd4545718ee15ca6c49"} Oct 30 22:13:17.194401 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:17.194250 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe3a13a56832aa2f396fcaf82bba09e204af67cc2a624cd4545718ee15ca6c49" Oct 30 22:13:17.194401 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:17.194208 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-69e4b2-62tk5" Oct 30 22:13:17.376523 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:17.376461 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v" Oct 30 22:13:17.468784 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:17.468750 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/024f007a-0f46-4093-8554-93a335039f10-istio-token\" (UniqueName: \"kubernetes.io/projected/024f007a-0f46-4093-8554-93a335039f10-istio-token\") pod \"024f007a-0f46-4093-8554-93a335039f10\" (UID: \"024f007a-0f46-4093-8554-93a335039f10\") " Oct 30 22:13:17.468996 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:17.468804 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/024f007a-0f46-4093-8554-93a335039f10-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/024f007a-0f46-4093-8554-93a335039f10-workload-certs\") pod \"024f007a-0f46-4093-8554-93a335039f10\" (UID: \"024f007a-0f46-4093-8554-93a335039f10\") " Oct 30 22:13:17.468996 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:17.468839 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/024f007a-0f46-4093-8554-93a335039f10-kube-api-access-9mvqz\" (UniqueName: \"kubernetes.io/projected/024f007a-0f46-4093-8554-93a335039f10-kube-api-access-9mvqz\") pod \"024f007a-0f46-4093-8554-93a335039f10\" (UID: \"024f007a-0f46-4093-8554-93a335039f10\") " Oct 30 22:13:17.468996 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:17.468872 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/024f007a-0f46-4093-8554-93a335039f10-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/024f007a-0f46-4093-8554-93a335039f10-istio-data\") pod \"024f007a-0f46-4093-8554-93a335039f10\" (UID: \"024f007a-0f46-4093-8554-93a335039f10\") " Oct 30 22:13:17.468996 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:17.468928 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/024f007a-0f46-4093-8554-93a335039f10-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/024f007a-0f46-4093-8554-93a335039f10-istio-podinfo\") pod \"024f007a-0f46-4093-8554-93a335039f10\" (UID: \"024f007a-0f46-4093-8554-93a335039f10\") " Oct 30 22:13:17.468996 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:17.468987 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/024f007a-0f46-4093-8554-93a335039f10-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/024f007a-0f46-4093-8554-93a335039f10-istio-envoy\") pod \"024f007a-0f46-4093-8554-93a335039f10\" (UID: \"024f007a-0f46-4093-8554-93a335039f10\") " Oct 30 22:13:17.469300 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:17.469055 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/024f007a-0f46-4093-8554-93a335039f10-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/024f007a-0f46-4093-8554-93a335039f10-istiod-ca-cert\") pod \"024f007a-0f46-4093-8554-93a335039f10\" (UID: \"024f007a-0f46-4093-8554-93a335039f10\") " Oct 30 22:13:17.469300 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:17.469089 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/024f007a-0f46-4093-8554-93a335039f10-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/024f007a-0f46-4093-8554-93a335039f10-workload-socket\") pod \"024f007a-0f46-4093-8554-93a335039f10\" (UID: \"024f007a-0f46-4093-8554-93a335039f10\") " Oct 30 22:13:17.469300 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:17.469169 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/024f007a-0f46-4093-8554-93a335039f10-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/024f007a-0f46-4093-8554-93a335039f10-credential-socket\") pod \"024f007a-0f46-4093-8554-93a335039f10\" (UID: \"024f007a-0f46-4093-8554-93a335039f10\") " Oct 30 22:13:17.469300 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:17.469204 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/024f007a-0f46-4093-8554-93a335039f10-workload-certs" pod "024f007a-0f46-4093-8554-93a335039f10" (UID: "024f007a-0f46-4093-8554-93a335039f10"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:17.469300 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:17.469265 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/024f007a-0f46-4093-8554-93a335039f10-istio-data" pod "024f007a-0f46-4093-8554-93a335039f10" (UID: "024f007a-0f46-4093-8554-93a335039f10"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:17.469630 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:17.469442 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/024f007a-0f46-4093-8554-93a335039f10-credential-socket" pod "024f007a-0f46-4093-8554-93a335039f10" (UID: "024f007a-0f46-4093-8554-93a335039f10"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:17.469630 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:17.469529 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/024f007a-0f46-4093-8554-93a335039f10-workload-socket" pod "024f007a-0f46-4093-8554-93a335039f10" (UID: "024f007a-0f46-4093-8554-93a335039f10"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:17.469630 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:17.469549 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/024f007a-0f46-4093-8554-93a335039f10-istiod-ca-cert" pod "024f007a-0f46-4093-8554-93a335039f10" (UID: "024f007a-0f46-4093-8554-93a335039f10"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:13:17.469892 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:17.469704 5748 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/024f007a-0f46-4093-8554-93a335039f10-workload-certs\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:17.469892 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:17.469724 5748 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/024f007a-0f46-4093-8554-93a335039f10-istio-data\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:17.471548 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:17.471527 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/024f007a-0f46-4093-8554-93a335039f10-kube-api-access-9mvqz" pod "024f007a-0f46-4093-8554-93a335039f10" (UID: "024f007a-0f46-4093-8554-93a335039f10"). InnerVolumeSpecName "kube-api-access-9mvqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:17.471548 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:17.471537 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/024f007a-0f46-4093-8554-93a335039f10-istio-podinfo" pod "024f007a-0f46-4093-8554-93a335039f10" (UID: "024f007a-0f46-4093-8554-93a335039f10"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:13:17.471677 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:17.471557 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/024f007a-0f46-4093-8554-93a335039f10-istio-token" pod "024f007a-0f46-4093-8554-93a335039f10" (UID: "024f007a-0f46-4093-8554-93a335039f10"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:17.471757 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:17.471739 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/024f007a-0f46-4093-8554-93a335039f10-istio-envoy" pod "024f007a-0f46-4093-8554-93a335039f10" (UID: "024f007a-0f46-4093-8554-93a335039f10"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:17.570803 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:17.570719 5748 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/024f007a-0f46-4093-8554-93a335039f10-istio-envoy\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:17.570803 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:17.570756 5748 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/024f007a-0f46-4093-8554-93a335039f10-istiod-ca-cert\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:17.570803 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:17.570771 5748 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/024f007a-0f46-4093-8554-93a335039f10-workload-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:17.570803 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:17.570785 5748 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/024f007a-0f46-4093-8554-93a335039f10-credential-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:17.570803 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:17.570799 5748 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/024f007a-0f46-4093-8554-93a335039f10-istio-token\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:17.570803 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:17.570812 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mvqz\" (UniqueName: \"kubernetes.io/projected/024f007a-0f46-4093-8554-93a335039f10-kube-api-access-9mvqz\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:17.571161 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:17.570824 5748 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/024f007a-0f46-4093-8554-93a335039f10-istio-podinfo\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:17.580879 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:17.580846 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-69e4b2/premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b"] Oct 30 22:13:17.592101 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:17.592071 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-69e4b2/simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj"] Oct 30 22:13:18.201581 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:18.201545 5748 generic.go:356] "Generic (PLEG): container finished" podID="024f007a-0f46-4093-8554-93a335039f10" containerID="a07053a9af92e329f37abf16ea55d304cf51778fe8301ed34cedca2613fe3319" exitCode=0 Oct 30 22:13:18.201781 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:18.201600 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v" Oct 30 22:13:18.201781 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:18.201634 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v" event={"ID":"024f007a-0f46-4093-8554-93a335039f10","Type":"ContainerDied","Data":"a07053a9af92e329f37abf16ea55d304cf51778fe8301ed34cedca2613fe3319"} Oct 30 22:13:18.201781 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:18.201689 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v" event={"ID":"024f007a-0f46-4093-8554-93a335039f10","Type":"ContainerDied","Data":"c38d058a1d57e95eca85b1add4fc61e2cb31e76766961d297d49e31bfdb66272"} Oct 30 22:13:18.201781 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:18.201713 5748 scope.go:122] "RemoveContainer" containerID="a07053a9af92e329f37abf16ea55d304cf51778fe8301ed34cedca2613fe3319" Oct 30 22:13:18.212514 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:18.212497 5748 scope.go:122] "RemoveContainer" containerID="6f596f846951d9760f05100004a2df05e925bb0ac478a90729c4d3a45d7c021a" Oct 30 22:13:18.220905 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:18.220878 5748 scope.go:122] "RemoveContainer" containerID="a07053a9af92e329f37abf16ea55d304cf51778fe8301ed34cedca2613fe3319" Oct 30 22:13:18.221170 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:13:18.221150 5748 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a07053a9af92e329f37abf16ea55d304cf51778fe8301ed34cedca2613fe3319\": container with ID starting with a07053a9af92e329f37abf16ea55d304cf51778fe8301ed34cedca2613fe3319 not found: ID does not exist" containerID="a07053a9af92e329f37abf16ea55d304cf51778fe8301ed34cedca2613fe3319" Oct 30 22:13:18.221240 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:18.221182 5748 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a07053a9af92e329f37abf16ea55d304cf51778fe8301ed34cedca2613fe3319"} err="failed to get container status \"a07053a9af92e329f37abf16ea55d304cf51778fe8301ed34cedca2613fe3319\": rpc error: code = NotFound desc = could not find container \"a07053a9af92e329f37abf16ea55d304cf51778fe8301ed34cedca2613fe3319\": container with ID starting with a07053a9af92e329f37abf16ea55d304cf51778fe8301ed34cedca2613fe3319 not found: ID does not exist" Oct 30 22:13:18.221240 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:18.221207 5748 scope.go:122] "RemoveContainer" containerID="6f596f846951d9760f05100004a2df05e925bb0ac478a90729c4d3a45d7c021a" Oct 30 22:13:18.221431 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:13:18.221412 5748 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f596f846951d9760f05100004a2df05e925bb0ac478a90729c4d3a45d7c021a\": container with ID starting with 6f596f846951d9760f05100004a2df05e925bb0ac478a90729c4d3a45d7c021a not found: ID does not exist" containerID="6f596f846951d9760f05100004a2df05e925bb0ac478a90729c4d3a45d7c021a" Oct 30 22:13:18.221475 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:18.221437 5748 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f596f846951d9760f05100004a2df05e925bb0ac478a90729c4d3a45d7c021a"} err="failed to get container status \"6f596f846951d9760f05100004a2df05e925bb0ac478a90729c4d3a45d7c021a\": rpc error: code = NotFound desc = could not find container \"6f596f846951d9760f05100004a2df05e925bb0ac478a90729c4d3a45d7c021a\": container with ID starting with 6f596f846951d9760f05100004a2df05e925bb0ac478a90729c4d3a45d7c021a not found: ID does not exist" Oct 30 22:13:18.368626 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:18.368584 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v"] Oct 30 22:13:18.419364 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:18.419323 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-19bc92-openshift-default-74b7bd6d97-f5l4v"] Oct 30 22:13:20.416917 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:20.416863 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="024f007a-0f46-4093-8554-93a335039f10" path="/var/lib/kubelet/pods/024f007a-0f46-4093-8554-93a335039f10/volumes" Oct 30 22:13:24.665492 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:24.665460 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-69e4b2-5845f89958-mzv58"] Oct 30 22:13:24.772212 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:24.772179 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-69e4b2-77c69fd5cf-5jr47"] Oct 30 22:13:27.289126 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:27.289092 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91bf7838-9vvk4"] Oct 30 22:13:27.289518 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:27.289498 5748 cpu_manager.go:414] "containerMap: removing container" podUID="e1370cc1-be39-4620-93c5-54d69fb6ef3b" containerName="revoke-keys" Oct 30 22:13:27.289518 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:27.289510 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1370cc1-be39-4620-93c5-54d69fb6ef3b" containerName="revoke-keys" Oct 30 22:13:27.289594 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:27.289533 5748 cpu_manager.go:414] "containerMap: removing container" podUID="024f007a-0f46-4093-8554-93a335039f10" containerName="istio-proxy" Oct 30 22:13:27.289594 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:27.289538 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="024f007a-0f46-4093-8554-93a335039f10" containerName="istio-proxy" Oct 30 22:13:27.289594 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:27.289544 5748 cpu_manager.go:414] "containerMap: removing container" podUID="024f007a-0f46-4093-8554-93a335039f10" containerName="istio-proxy" Oct 30 22:13:27.289594 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:27.289549 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="024f007a-0f46-4093-8554-93a335039f10" containerName="istio-proxy" Oct 30 22:13:27.289769 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:27.289643 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="024f007a-0f46-4093-8554-93a335039f10" containerName="istio-proxy" Oct 30 22:13:27.289769 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:27.289652 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="024f007a-0f46-4093-8554-93a335039f10" containerName="istio-proxy" Oct 30 22:13:27.289769 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:27.289664 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1370cc1-be39-4620-93c5-54d69fb6ef3b" containerName="revoke-keys" Oct 30 22:13:27.294391 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:27.294370 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91bf7838-9vvk4" Oct 30 22:13:27.347696 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:27.347663 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:13:27.347869 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:27.347707 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:13:27.398221 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:27.398194 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:13:27.398395 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:27.398241 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 22:13:27.400515 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:27.400490 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91bf7838-9vvk4"] Oct 30 22:13:27.464076 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:27.464049 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92547c4c-d9ff-4b5b-a8f5-dff883c46899-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91bf7838-9vvk4\" (UID: \"92547c4c-d9ff-4b5b-a8f5-dff883c46899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91bf7838-9vvk4" Oct 30 22:13:27.464232 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:27.464185 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2gqb\" (UniqueName: \"kubernetes.io/projected/92547c4c-d9ff-4b5b-a8f5-dff883c46899-kube-api-access-v2gqb\") pod \"maas-api-revoke-sub-91bf7838-9vvk4\" (UID: \"92547c4c-d9ff-4b5b-a8f5-dff883c46899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91bf7838-9vvk4" Oct 30 22:13:27.464232 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:27.464227 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92547c4c-d9ff-4b5b-a8f5-dff883c46899-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91bf7838-9vvk4\" (UID: \"92547c4c-d9ff-4b5b-a8f5-dff883c46899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91bf7838-9vvk4" Oct 30 22:13:27.565196 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:27.565130 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2gqb\" (UniqueName: \"kubernetes.io/projected/92547c4c-d9ff-4b5b-a8f5-dff883c46899-kube-api-access-v2gqb\") pod \"maas-api-revoke-sub-91bf7838-9vvk4\" (UID: \"92547c4c-d9ff-4b5b-a8f5-dff883c46899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91bf7838-9vvk4" Oct 30 22:13:27.565196 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:27.565170 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92547c4c-d9ff-4b5b-a8f5-dff883c46899-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91bf7838-9vvk4\" (UID: \"92547c4c-d9ff-4b5b-a8f5-dff883c46899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91bf7838-9vvk4" Oct 30 22:13:27.565404 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:27.565246 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92547c4c-d9ff-4b5b-a8f5-dff883c46899-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91bf7838-9vvk4\" (UID: \"92547c4c-d9ff-4b5b-a8f5-dff883c46899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91bf7838-9vvk4" Oct 30 22:13:27.565875 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:27.565849 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92547c4c-d9ff-4b5b-a8f5-dff883c46899-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91bf7838-9vvk4\" (UID: \"92547c4c-d9ff-4b5b-a8f5-dff883c46899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91bf7838-9vvk4" Oct 30 22:13:27.567577 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:27.567554 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92547c4c-d9ff-4b5b-a8f5-dff883c46899-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91bf7838-9vvk4\" (UID: \"92547c4c-d9ff-4b5b-a8f5-dff883c46899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91bf7838-9vvk4" Oct 30 22:13:27.627225 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:27.627197 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2gqb\" (UniqueName: \"kubernetes.io/projected/92547c4c-d9ff-4b5b-a8f5-dff883c46899-kube-api-access-v2gqb\") pod \"maas-api-revoke-sub-91bf7838-9vvk4\" (UID: \"92547c4c-d9ff-4b5b-a8f5-dff883c46899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91bf7838-9vvk4" Oct 30 22:13:27.907833 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:27.907804 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91bf7838-9vvk4" Oct 30 22:13:28.288717 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:28.288685 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91bf7838-9vvk4"] Oct 30 22:13:28.290821 ip-10-0-143-61 kubenswrapper[5748]: W1030 22:13:28.290796 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92547c4c_d9ff_4b5b_a8f5_dff883c46899.slice/crio-896060abd21d1306a2dd2f958f934b9b3ff0184011aba7837928ef0ad83453cc WatchSource:0}: Error finding container 896060abd21d1306a2dd2f958f934b9b3ff0184011aba7837928ef0ad83453cc: Status 404 returned error can't find the container with id 896060abd21d1306a2dd2f958f934b9b3ff0184011aba7837928ef0ad83453cc Oct 30 22:13:29.266620 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:29.266585 5748 generic.go:356] "Generic (PLEG): container finished" podID="92547c4c-d9ff-4b5b-a8f5-dff883c46899" containerID="aa018b4ba5c912de99ce7983bcc47a2c41d3af64e0706d0f7e94548d08f63786" exitCode=0 Oct 30 22:13:29.266772 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:29.266665 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91bf7838-9vvk4" event={"ID":"92547c4c-d9ff-4b5b-a8f5-dff883c46899","Type":"ContainerDied","Data":"aa018b4ba5c912de99ce7983bcc47a2c41d3af64e0706d0f7e94548d08f63786"} Oct 30 22:13:29.266772 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:29.266716 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91bf7838-9vvk4" event={"ID":"92547c4c-d9ff-4b5b-a8f5-dff883c46899","Type":"ContainerStarted","Data":"896060abd21d1306a2dd2f958f934b9b3ff0184011aba7837928ef0ad83453cc"} Oct 30 22:13:29.666630 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:29.666592 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-69e4b2-5845f89958-mzv58" podUID="1ecabb08-132c-4b22-b550-77dbff23555d" containerName="payload-processing" containerID="cri-o://ef30e8cfa3a3c4963cebc0ebca7f1785ad17713ae417a3ba87292630051a036c" gracePeriod=25 Oct 30 22:13:29.772778 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:29.772724 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-69e4b2-77c69fd5cf-5jr47" podUID="08366fbc-97d1-41a0-959a-447d00b7ec88" containerName="payload-pre-processing" containerID="cri-o://f69d487bf6d71e9461558b17030c95f7755742631a35751108b038a6aa1c5f76" gracePeriod=25 Oct 30 22:13:30.143932 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:13:30.143877 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:13:30.406967 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:30.406911 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91bf7838-9vvk4" Oct 30 22:13:30.490604 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:30.490573 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92547c4c-d9ff-4b5b-a8f5-dff883c46899-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92547c4c-d9ff-4b5b-a8f5-dff883c46899-maas-api-xks-ca\") pod \"92547c4c-d9ff-4b5b-a8f5-dff883c46899\" (UID: \"92547c4c-d9ff-4b5b-a8f5-dff883c46899\") " Oct 30 22:13:30.490779 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:30.490695 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92547c4c-d9ff-4b5b-a8f5-dff883c46899-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92547c4c-d9ff-4b5b-a8f5-dff883c46899-maas-api-openshift-ca\") pod \"92547c4c-d9ff-4b5b-a8f5-dff883c46899\" (UID: \"92547c4c-d9ff-4b5b-a8f5-dff883c46899\") " Oct 30 22:13:30.490779 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:30.490760 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92547c4c-d9ff-4b5b-a8f5-dff883c46899-kube-api-access-v2gqb\" (UniqueName: \"kubernetes.io/projected/92547c4c-d9ff-4b5b-a8f5-dff883c46899-kube-api-access-v2gqb\") pod \"92547c4c-d9ff-4b5b-a8f5-dff883c46899\" (UID: \"92547c4c-d9ff-4b5b-a8f5-dff883c46899\") " Oct 30 22:13:30.491095 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:30.491074 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92547c4c-d9ff-4b5b-a8f5-dff883c46899-maas-api-openshift-ca" pod "92547c4c-d9ff-4b5b-a8f5-dff883c46899" (UID: "92547c4c-d9ff-4b5b-a8f5-dff883c46899"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:13:30.492641 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:30.492621 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92547c4c-d9ff-4b5b-a8f5-dff883c46899-maas-api-xks-ca" pod "92547c4c-d9ff-4b5b-a8f5-dff883c46899" (UID: "92547c4c-d9ff-4b5b-a8f5-dff883c46899"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:13:30.492701 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:30.492673 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92547c4c-d9ff-4b5b-a8f5-dff883c46899-kube-api-access-v2gqb" pod "92547c4c-d9ff-4b5b-a8f5-dff883c46899" (UID: "92547c4c-d9ff-4b5b-a8f5-dff883c46899"). InnerVolumeSpecName "kube-api-access-v2gqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:30.592355 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:30.592324 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92547c4c-d9ff-4b5b-a8f5-dff883c46899-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:30.592525 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:30.592362 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2gqb\" (UniqueName: \"kubernetes.io/projected/92547c4c-d9ff-4b5b-a8f5-dff883c46899-kube-api-access-v2gqb\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:30.592525 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:30.592375 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92547c4c-d9ff-4b5b-a8f5-dff883c46899-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:31.279050 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:31.279015 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91bf7838-9vvk4" Oct 30 22:13:31.279513 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:31.279057 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91bf7838-9vvk4" event={"ID":"92547c4c-d9ff-4b5b-a8f5-dff883c46899","Type":"ContainerDied","Data":"896060abd21d1306a2dd2f958f934b9b3ff0184011aba7837928ef0ad83453cc"} Oct 30 22:13:31.279513 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:31.279101 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="896060abd21d1306a2dd2f958f934b9b3ff0184011aba7837928ef0ad83453cc" Oct 30 22:13:32.601018 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:32.600954 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-69e4b2/premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b" podUID="3ed43145-33bd-4c11-bb29-dc852511f6d2" containerName="main" containerID="cri-o://4244e067e8d04b554524e70d4bfe61ef05f6d4cb815bf2c2e2c85ed425897201" gracePeriod=45 Oct 30 22:13:32.611304 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:32.611272 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-69e4b2/simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj" podUID="698fc0f2-64ce-4e7e-b482-8ebd9f33f84d" containerName="main" containerID="cri-o://75ff557b501509dd25ee3faab1db2ba7d2b25e88e5954f1f97ae72231999acce" gracePeriod=45 Oct 30 22:13:32.750683 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:32.750637 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-69e4b2/premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b" podUID="3ed43145-33bd-4c11-bb29-dc852511f6d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": dial tcp 10.134.0.47:8000: connect: connection refused" Oct 30 22:13:32.776604 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:32.776556 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-69e4b2/simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj" podUID="698fc0f2-64ce-4e7e-b482-8ebd9f33f84d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": dial tcp 10.134.0.44:8000: connect: connection refused" Oct 30 22:13:32.859547 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:32.859483 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-69e4b2/premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b" Oct 30 22:13:32.887283 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:32.887260 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-69e4b2/simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj" Oct 30 22:13:32.911903 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:32.911878 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/698fc0f2-64ce-4e7e-b482-8ebd9f33f84d-kube-api-access-ndhwn\" (UniqueName: \"kubernetes.io/projected/698fc0f2-64ce-4e7e-b482-8ebd9f33f84d-kube-api-access-ndhwn\") pod \"698fc0f2-64ce-4e7e-b482-8ebd9f33f84d\" (UID: \"698fc0f2-64ce-4e7e-b482-8ebd9f33f84d\") " Oct 30 22:13:32.912032 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:32.911933 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/698fc0f2-64ce-4e7e-b482-8ebd9f33f84d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/698fc0f2-64ce-4e7e-b482-8ebd9f33f84d-model-cache\") pod \"698fc0f2-64ce-4e7e-b482-8ebd9f33f84d\" (UID: \"698fc0f2-64ce-4e7e-b482-8ebd9f33f84d\") " Oct 30 22:13:32.912032 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:32.911986 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ed43145-33bd-4c11-bb29-dc852511f6d2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ed43145-33bd-4c11-bb29-dc852511f6d2-model-cache\") pod \"3ed43145-33bd-4c11-bb29-dc852511f6d2\" (UID: \"3ed43145-33bd-4c11-bb29-dc852511f6d2\") " Oct 30 22:13:32.912032 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:32.912012 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ed43145-33bd-4c11-bb29-dc852511f6d2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ed43145-33bd-4c11-bb29-dc852511f6d2-dshm\") pod \"3ed43145-33bd-4c11-bb29-dc852511f6d2\" (UID: \"3ed43145-33bd-4c11-bb29-dc852511f6d2\") " Oct 30 22:13:32.912210 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:32.912043 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/698fc0f2-64ce-4e7e-b482-8ebd9f33f84d-home\" (UniqueName: \"kubernetes.io/empty-dir/698fc0f2-64ce-4e7e-b482-8ebd9f33f84d-home\") pod \"698fc0f2-64ce-4e7e-b482-8ebd9f33f84d\" (UID: \"698fc0f2-64ce-4e7e-b482-8ebd9f33f84d\") " Oct 30 22:13:32.912210 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:32.912092 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/698fc0f2-64ce-4e7e-b482-8ebd9f33f84d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/698fc0f2-64ce-4e7e-b482-8ebd9f33f84d-tmp-dir\") pod \"698fc0f2-64ce-4e7e-b482-8ebd9f33f84d\" (UID: \"698fc0f2-64ce-4e7e-b482-8ebd9f33f84d\") " Oct 30 22:13:32.912210 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:32.912139 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ed43145-33bd-4c11-bb29-dc852511f6d2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ed43145-33bd-4c11-bb29-dc852511f6d2-tmp-dir\") pod \"3ed43145-33bd-4c11-bb29-dc852511f6d2\" (UID: \"3ed43145-33bd-4c11-bb29-dc852511f6d2\") " Oct 30 22:13:32.912210 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:32.912171 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ed43145-33bd-4c11-bb29-dc852511f6d2-tls-certs\" (UniqueName: \"kubernetes.io/secret/3ed43145-33bd-4c11-bb29-dc852511f6d2-tls-certs\") pod \"3ed43145-33bd-4c11-bb29-dc852511f6d2\" (UID: \"3ed43145-33bd-4c11-bb29-dc852511f6d2\") " Oct 30 22:13:32.912210 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:32.912209 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/698fc0f2-64ce-4e7e-b482-8ebd9f33f84d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/698fc0f2-64ce-4e7e-b482-8ebd9f33f84d-dshm\") pod \"698fc0f2-64ce-4e7e-b482-8ebd9f33f84d\" (UID: \"698fc0f2-64ce-4e7e-b482-8ebd9f33f84d\") " Oct 30 22:13:32.912471 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:32.912261 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ed43145-33bd-4c11-bb29-dc852511f6d2-home\" (UniqueName: \"kubernetes.io/empty-dir/3ed43145-33bd-4c11-bb29-dc852511f6d2-home\") pod \"3ed43145-33bd-4c11-bb29-dc852511f6d2\" (UID: \"3ed43145-33bd-4c11-bb29-dc852511f6d2\") " Oct 30 22:13:32.912471 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:32.912270 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ed43145-33bd-4c11-bb29-dc852511f6d2-model-cache" pod "3ed43145-33bd-4c11-bb29-dc852511f6d2" (UID: "3ed43145-33bd-4c11-bb29-dc852511f6d2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:32.912471 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:32.912290 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ed43145-33bd-4c11-bb29-dc852511f6d2-kube-api-access-dqnkt\" (UniqueName: \"kubernetes.io/projected/3ed43145-33bd-4c11-bb29-dc852511f6d2-kube-api-access-dqnkt\") pod \"3ed43145-33bd-4c11-bb29-dc852511f6d2\" (UID: \"3ed43145-33bd-4c11-bb29-dc852511f6d2\") " Oct 30 22:13:32.912471 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:32.912278 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/698fc0f2-64ce-4e7e-b482-8ebd9f33f84d-model-cache" pod "698fc0f2-64ce-4e7e-b482-8ebd9f33f84d" (UID: "698fc0f2-64ce-4e7e-b482-8ebd9f33f84d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:32.912471 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:32.912329 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/698fc0f2-64ce-4e7e-b482-8ebd9f33f84d-tls-certs\" (UniqueName: \"kubernetes.io/secret/698fc0f2-64ce-4e7e-b482-8ebd9f33f84d-tls-certs\") pod \"698fc0f2-64ce-4e7e-b482-8ebd9f33f84d\" (UID: \"698fc0f2-64ce-4e7e-b482-8ebd9f33f84d\") " Oct 30 22:13:32.912471 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:32.912458 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ed43145-33bd-4c11-bb29-dc852511f6d2-tmp-dir" pod "3ed43145-33bd-4c11-bb29-dc852511f6d2" (UID: "3ed43145-33bd-4c11-bb29-dc852511f6d2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:32.912827 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:32.912489 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ed43145-33bd-4c11-bb29-dc852511f6d2-home" pod "3ed43145-33bd-4c11-bb29-dc852511f6d2" (UID: "3ed43145-33bd-4c11-bb29-dc852511f6d2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:32.912827 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:32.912645 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/698fc0f2-64ce-4e7e-b482-8ebd9f33f84d-home" pod "698fc0f2-64ce-4e7e-b482-8ebd9f33f84d" (UID: "698fc0f2-64ce-4e7e-b482-8ebd9f33f84d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:32.912999 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:32.912836 5748 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/698fc0f2-64ce-4e7e-b482-8ebd9f33f84d-model-cache\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:32.912999 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:32.912854 5748 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ed43145-33bd-4c11-bb29-dc852511f6d2-model-cache\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:32.912999 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:32.912866 5748 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/698fc0f2-64ce-4e7e-b482-8ebd9f33f84d-home\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:32.912999 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:32.912880 5748 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ed43145-33bd-4c11-bb29-dc852511f6d2-tmp-dir\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:32.912999 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:32.912891 5748 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ed43145-33bd-4c11-bb29-dc852511f6d2-home\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:32.913268 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:32.913110 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/698fc0f2-64ce-4e7e-b482-8ebd9f33f84d-tmp-dir" pod "698fc0f2-64ce-4e7e-b482-8ebd9f33f84d" (UID: "698fc0f2-64ce-4e7e-b482-8ebd9f33f84d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:32.921070 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:32.921036 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ed43145-33bd-4c11-bb29-dc852511f6d2-dshm" pod "3ed43145-33bd-4c11-bb29-dc852511f6d2" (UID: "3ed43145-33bd-4c11-bb29-dc852511f6d2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:32.921200 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:32.921109 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/698fc0f2-64ce-4e7e-b482-8ebd9f33f84d-dshm" pod "698fc0f2-64ce-4e7e-b482-8ebd9f33f84d" (UID: "698fc0f2-64ce-4e7e-b482-8ebd9f33f84d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:32.921200 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:32.921117 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ed43145-33bd-4c11-bb29-dc852511f6d2-kube-api-access-dqnkt" pod "3ed43145-33bd-4c11-bb29-dc852511f6d2" (UID: "3ed43145-33bd-4c11-bb29-dc852511f6d2"). InnerVolumeSpecName "kube-api-access-dqnkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:32.921200 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:32.921127 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/698fc0f2-64ce-4e7e-b482-8ebd9f33f84d-tls-certs" pod "698fc0f2-64ce-4e7e-b482-8ebd9f33f84d" (UID: "698fc0f2-64ce-4e7e-b482-8ebd9f33f84d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:13:32.921200 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:32.921156 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/698fc0f2-64ce-4e7e-b482-8ebd9f33f84d-kube-api-access-ndhwn" pod "698fc0f2-64ce-4e7e-b482-8ebd9f33f84d" (UID: "698fc0f2-64ce-4e7e-b482-8ebd9f33f84d"). InnerVolumeSpecName "kube-api-access-ndhwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:32.921200 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:32.921169 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ed43145-33bd-4c11-bb29-dc852511f6d2-tls-certs" pod "3ed43145-33bd-4c11-bb29-dc852511f6d2" (UID: "3ed43145-33bd-4c11-bb29-dc852511f6d2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:13:33.014544 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:33.014504 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndhwn\" (UniqueName: \"kubernetes.io/projected/698fc0f2-64ce-4e7e-b482-8ebd9f33f84d-kube-api-access-ndhwn\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:33.014707 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:33.014548 5748 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ed43145-33bd-4c11-bb29-dc852511f6d2-dshm\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:33.014707 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:33.014592 5748 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/698fc0f2-64ce-4e7e-b482-8ebd9f33f84d-tmp-dir\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:33.014707 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:33.014608 5748 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ed43145-33bd-4c11-bb29-dc852511f6d2-tls-certs\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:33.014707 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:33.014622 5748 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/698fc0f2-64ce-4e7e-b482-8ebd9f33f84d-dshm\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:33.014707 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:33.014635 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqnkt\" (UniqueName: \"kubernetes.io/projected/3ed43145-33bd-4c11-bb29-dc852511f6d2-kube-api-access-dqnkt\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:33.014707 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:33.014650 5748 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/698fc0f2-64ce-4e7e-b482-8ebd9f33f84d-tls-certs\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:33.291924 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:33.291873 5748 generic.go:356] "Generic (PLEG): container finished" podID="698fc0f2-64ce-4e7e-b482-8ebd9f33f84d" containerID="75ff557b501509dd25ee3faab1db2ba7d2b25e88e5954f1f97ae72231999acce" exitCode=0 Oct 30 22:13:33.292119 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:33.291955 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-69e4b2/simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj" Oct 30 22:13:33.292119 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:33.291957 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-69e4b2/simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj" event={"ID":"698fc0f2-64ce-4e7e-b482-8ebd9f33f84d","Type":"ContainerDied","Data":"75ff557b501509dd25ee3faab1db2ba7d2b25e88e5954f1f97ae72231999acce"} Oct 30 22:13:33.292119 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:33.292004 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-69e4b2/simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj" event={"ID":"698fc0f2-64ce-4e7e-b482-8ebd9f33f84d","Type":"ContainerDied","Data":"ca0f1ae32cd917020a6b1229fb468ef67eeb0acdc5e2c4fb524f0ce5c7c7b2fb"} Oct 30 22:13:33.292119 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:33.292022 5748 scope.go:122] "RemoveContainer" containerID="75ff557b501509dd25ee3faab1db2ba7d2b25e88e5954f1f97ae72231999acce" Oct 30 22:13:33.293553 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:33.293525 5748 generic.go:356] "Generic (PLEG): container finished" podID="3ed43145-33bd-4c11-bb29-dc852511f6d2" containerID="4244e067e8d04b554524e70d4bfe61ef05f6d4cb815bf2c2e2c85ed425897201" exitCode=0 Oct 30 22:13:33.293676 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:33.293560 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-69e4b2/premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b" event={"ID":"3ed43145-33bd-4c11-bb29-dc852511f6d2","Type":"ContainerDied","Data":"4244e067e8d04b554524e70d4bfe61ef05f6d4cb815bf2c2e2c85ed425897201"} Oct 30 22:13:33.293676 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:33.293601 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-69e4b2/premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b" Oct 30 22:13:33.293676 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:33.293611 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-69e4b2/premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b" event={"ID":"3ed43145-33bd-4c11-bb29-dc852511f6d2","Type":"ContainerDied","Data":"dd75240afec164be7bf5ee1f8a71bfa76c29e79d92a3081a1e0cb30093e9b272"} Oct 30 22:13:33.302793 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:33.302771 5748 scope.go:122] "RemoveContainer" containerID="75ff557b501509dd25ee3faab1db2ba7d2b25e88e5954f1f97ae72231999acce" Oct 30 22:13:33.303077 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:13:33.303057 5748 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75ff557b501509dd25ee3faab1db2ba7d2b25e88e5954f1f97ae72231999acce\": container with ID starting with 75ff557b501509dd25ee3faab1db2ba7d2b25e88e5954f1f97ae72231999acce not found: ID does not exist" containerID="75ff557b501509dd25ee3faab1db2ba7d2b25e88e5954f1f97ae72231999acce" Oct 30 22:13:33.303172 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:33.303085 5748 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75ff557b501509dd25ee3faab1db2ba7d2b25e88e5954f1f97ae72231999acce"} err="failed to get container status \"75ff557b501509dd25ee3faab1db2ba7d2b25e88e5954f1f97ae72231999acce\": rpc error: code = NotFound desc = could not find container \"75ff557b501509dd25ee3faab1db2ba7d2b25e88e5954f1f97ae72231999acce\": container with ID starting with 75ff557b501509dd25ee3faab1db2ba7d2b25e88e5954f1f97ae72231999acce not found: ID does not exist" Oct 30 22:13:33.303172 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:33.303101 5748 scope.go:122] "RemoveContainer" containerID="4244e067e8d04b554524e70d4bfe61ef05f6d4cb815bf2c2e2c85ed425897201" Oct 30 22:13:33.311217 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:33.311201 5748 scope.go:122] "RemoveContainer" containerID="4244e067e8d04b554524e70d4bfe61ef05f6d4cb815bf2c2e2c85ed425897201" Oct 30 22:13:33.311457 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:13:33.311438 5748 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4244e067e8d04b554524e70d4bfe61ef05f6d4cb815bf2c2e2c85ed425897201\": container with ID starting with 4244e067e8d04b554524e70d4bfe61ef05f6d4cb815bf2c2e2c85ed425897201 not found: ID does not exist" containerID="4244e067e8d04b554524e70d4bfe61ef05f6d4cb815bf2c2e2c85ed425897201" Oct 30 22:13:33.311503 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:33.311466 5748 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4244e067e8d04b554524e70d4bfe61ef05f6d4cb815bf2c2e2c85ed425897201"} err="failed to get container status \"4244e067e8d04b554524e70d4bfe61ef05f6d4cb815bf2c2e2c85ed425897201\": rpc error: code = NotFound desc = could not find container \"4244e067e8d04b554524e70d4bfe61ef05f6d4cb815bf2c2e2c85ed425897201\": container with ID starting with 4244e067e8d04b554524e70d4bfe61ef05f6d4cb815bf2c2e2c85ed425897201 not found: ID does not exist" Oct 30 22:13:33.458059 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:33.458027 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-69e4b2/premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b"] Oct 30 22:13:33.462196 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:33.462169 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-69e4b2/premium-w6-69e4b2-kserve-b4d66cdd8-ncf5b"] Oct 30 22:13:33.572607 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:33.572537 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-69e4b2/simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj"] Oct 30 22:13:33.578687 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:33.578661 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-69e4b2/simulator-w6-69e4b2-kserve-8bd6fc957-jz6hj"] Oct 30 22:13:34.415569 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:34.415528 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ed43145-33bd-4c11-bb29-dc852511f6d2" path="/var/lib/kubelet/pods/3ed43145-33bd-4c11-bb29-dc852511f6d2/volumes" Oct 30 22:13:34.416003 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:34.415987 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="698fc0f2-64ce-4e7e-b482-8ebd9f33f84d" path="/var/lib/kubelet/pods/698fc0f2-64ce-4e7e-b482-8ebd9f33f84d/volumes" Oct 30 22:13:34.502705 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:34.502667 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-69e4b2-5845f89958-mzv58" podUID="1ecabb08-132c-4b22-b550-77dbff23555d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:13:34.502705 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:34.502691 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-69e4b2-77c69fd5cf-5jr47" podUID="08366fbc-97d1-41a0-959a-447d00b7ec88" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:13:36.911373 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:36.911344 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j4v8r_3df036ff-1c00-4ce2-9e86-8acbeedf6c82/ovn-acl-logging/0.log" Oct 30 22:13:36.924759 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:36.924736 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j4v8r_3df036ff-1c00-4ce2-9e86-8acbeedf6c82/ovn-acl-logging/0.log" Oct 30 22:13:38.493761 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:38.493729 5748 scope.go:122] "RemoveContainer" containerID="57ff2698c6cf57752bd9a4ee0e7661daa9fa1535b469427538de031b8e27c1fa" Oct 30 22:13:44.502803 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:44.502768 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-69e4b2-77c69fd5cf-5jr47" podUID="08366fbc-97d1-41a0-959a-447d00b7ec88" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:13:44.503227 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:44.502768 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-69e4b2-5845f89958-mzv58" podUID="1ecabb08-132c-4b22-b550-77dbff23555d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:13:44.762880 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:44.762796 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-69e4b2-62tk5"] Oct 30 22:13:44.813276 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:44.813238 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-69e4b2-62tk5"] Oct 30 22:13:46.416378 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:46.416344 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1370cc1-be39-4620-93c5-54d69fb6ef3b" path="/var/lib/kubelet/pods/e1370cc1-be39-4620-93c5-54d69fb6ef3b/volumes" Oct 30 22:13:49.914298 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:49.914269 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-69e4b2-5845f89958-mzv58" Oct 30 22:13:49.973700 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:49.973667 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ecabb08-132c-4b22-b550-77dbff23555d-kube-api-access-rnpmg\" (UniqueName: \"kubernetes.io/projected/1ecabb08-132c-4b22-b550-77dbff23555d-kube-api-access-rnpmg\") pod \"1ecabb08-132c-4b22-b550-77dbff23555d\" (UID: \"1ecabb08-132c-4b22-b550-77dbff23555d\") " Oct 30 22:13:49.973866 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:49.973748 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ecabb08-132c-4b22-b550-77dbff23555d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ecabb08-132c-4b22-b550-77dbff23555d-plugins-config-volume\") pod \"1ecabb08-132c-4b22-b550-77dbff23555d\" (UID: \"1ecabb08-132c-4b22-b550-77dbff23555d\") " Oct 30 22:13:49.974444 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:49.974394 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ecabb08-132c-4b22-b550-77dbff23555d-plugins-config-volume" pod "1ecabb08-132c-4b22-b550-77dbff23555d" (UID: "1ecabb08-132c-4b22-b550-77dbff23555d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:13:49.977198 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:49.977170 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ecabb08-132c-4b22-b550-77dbff23555d-kube-api-access-rnpmg" pod "1ecabb08-132c-4b22-b550-77dbff23555d" (UID: "1ecabb08-132c-4b22-b550-77dbff23555d"). InnerVolumeSpecName "kube-api-access-rnpmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:50.006408 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:50.006388 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-69e4b2-77c69fd5cf-5jr47" Oct 30 22:13:50.075253 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:50.075218 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08366fbc-97d1-41a0-959a-447d00b7ec88-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08366fbc-97d1-41a0-959a-447d00b7ec88-plugins-config-volume\") pod \"08366fbc-97d1-41a0-959a-447d00b7ec88\" (UID: \"08366fbc-97d1-41a0-959a-447d00b7ec88\") " Oct 30 22:13:50.075436 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:50.075356 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08366fbc-97d1-41a0-959a-447d00b7ec88-kube-api-access-ft9xz\" (UniqueName: \"kubernetes.io/projected/08366fbc-97d1-41a0-959a-447d00b7ec88-kube-api-access-ft9xz\") pod \"08366fbc-97d1-41a0-959a-447d00b7ec88\" (UID: \"08366fbc-97d1-41a0-959a-447d00b7ec88\") " Oct 30 22:13:50.075594 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:50.075570 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08366fbc-97d1-41a0-959a-447d00b7ec88-plugins-config-volume" pod "08366fbc-97d1-41a0-959a-447d00b7ec88" (UID: "08366fbc-97d1-41a0-959a-447d00b7ec88"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:13:50.075666 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:50.075643 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnpmg\" (UniqueName: \"kubernetes.io/projected/1ecabb08-132c-4b22-b550-77dbff23555d-kube-api-access-rnpmg\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:50.075715 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:50.075664 5748 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ecabb08-132c-4b22-b550-77dbff23555d-plugins-config-volume\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:50.077401 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:50.077380 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08366fbc-97d1-41a0-959a-447d00b7ec88-kube-api-access-ft9xz" pod "08366fbc-97d1-41a0-959a-447d00b7ec88" (UID: "08366fbc-97d1-41a0-959a-447d00b7ec88"). InnerVolumeSpecName "kube-api-access-ft9xz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:50.176835 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:50.176804 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ft9xz\" (UniqueName: \"kubernetes.io/projected/08366fbc-97d1-41a0-959a-447d00b7ec88-kube-api-access-ft9xz\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:50.176835 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:50.176834 5748 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08366fbc-97d1-41a0-959a-447d00b7ec88-plugins-config-volume\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:50.394368 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:50.394332 5748 generic.go:356] "Generic (PLEG): container finished" podID="08366fbc-97d1-41a0-959a-447d00b7ec88" containerID="f69d487bf6d71e9461558b17030c95f7755742631a35751108b038a6aa1c5f76" exitCode=0 Oct 30 22:13:50.394589 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:50.394373 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-69e4b2-77c69fd5cf-5jr47" event={"ID":"08366fbc-97d1-41a0-959a-447d00b7ec88","Type":"ContainerDied","Data":"f69d487bf6d71e9461558b17030c95f7755742631a35751108b038a6aa1c5f76"} Oct 30 22:13:50.394589 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:50.394394 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-69e4b2-77c69fd5cf-5jr47" Oct 30 22:13:50.394589 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:50.394469 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-69e4b2-77c69fd5cf-5jr47" event={"ID":"08366fbc-97d1-41a0-959a-447d00b7ec88","Type":"ContainerDied","Data":"5d02311587f0f72c092f74bbb60525b77945ec37872c669f0cba37de2fccf529"} Oct 30 22:13:50.394589 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:50.394493 5748 scope.go:122] "RemoveContainer" containerID="f69d487bf6d71e9461558b17030c95f7755742631a35751108b038a6aa1c5f76" Oct 30 22:13:50.397218 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:50.397126 5748 generic.go:356] "Generic (PLEG): container finished" podID="1ecabb08-132c-4b22-b550-77dbff23555d" containerID="ef30e8cfa3a3c4963cebc0ebca7f1785ad17713ae417a3ba87292630051a036c" exitCode=0 Oct 30 22:13:50.397341 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:50.397238 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-69e4b2-5845f89958-mzv58" event={"ID":"1ecabb08-132c-4b22-b550-77dbff23555d","Type":"ContainerDied","Data":"ef30e8cfa3a3c4963cebc0ebca7f1785ad17713ae417a3ba87292630051a036c"} Oct 30 22:13:50.397341 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:50.397282 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-69e4b2-5845f89958-mzv58" event={"ID":"1ecabb08-132c-4b22-b550-77dbff23555d","Type":"ContainerDied","Data":"f97eededf4b894e1e40ff465f99b77a0253351cdb295955304149586765bbbe8"} Oct 30 22:13:50.397341 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:50.397317 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-69e4b2-5845f89958-mzv58" Oct 30 22:13:50.407927 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:50.407875 5748 scope.go:122] "RemoveContainer" containerID="f69d487bf6d71e9461558b17030c95f7755742631a35751108b038a6aa1c5f76" Oct 30 22:13:50.408420 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:13:50.408395 5748 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f69d487bf6d71e9461558b17030c95f7755742631a35751108b038a6aa1c5f76\": container with ID starting with f69d487bf6d71e9461558b17030c95f7755742631a35751108b038a6aa1c5f76 not found: ID does not exist" containerID="f69d487bf6d71e9461558b17030c95f7755742631a35751108b038a6aa1c5f76" Oct 30 22:13:50.408539 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:50.408427 5748 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f69d487bf6d71e9461558b17030c95f7755742631a35751108b038a6aa1c5f76"} err="failed to get container status \"f69d487bf6d71e9461558b17030c95f7755742631a35751108b038a6aa1c5f76\": rpc error: code = NotFound desc = could not find container \"f69d487bf6d71e9461558b17030c95f7755742631a35751108b038a6aa1c5f76\": container with ID starting with f69d487bf6d71e9461558b17030c95f7755742631a35751108b038a6aa1c5f76 not found: ID does not exist" Oct 30 22:13:50.408539 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:50.408447 5748 scope.go:122] "RemoveContainer" containerID="ef30e8cfa3a3c4963cebc0ebca7f1785ad17713ae417a3ba87292630051a036c" Oct 30 22:13:50.421014 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:50.420994 5748 scope.go:122] "RemoveContainer" containerID="ef30e8cfa3a3c4963cebc0ebca7f1785ad17713ae417a3ba87292630051a036c" Oct 30 22:13:50.421310 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:13:50.421289 5748 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef30e8cfa3a3c4963cebc0ebca7f1785ad17713ae417a3ba87292630051a036c\": container with ID starting with ef30e8cfa3a3c4963cebc0ebca7f1785ad17713ae417a3ba87292630051a036c not found: ID does not exist" containerID="ef30e8cfa3a3c4963cebc0ebca7f1785ad17713ae417a3ba87292630051a036c" Oct 30 22:13:50.421389 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:50.421321 5748 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef30e8cfa3a3c4963cebc0ebca7f1785ad17713ae417a3ba87292630051a036c"} err="failed to get container status \"ef30e8cfa3a3c4963cebc0ebca7f1785ad17713ae417a3ba87292630051a036c\": rpc error: code = NotFound desc = could not find container \"ef30e8cfa3a3c4963cebc0ebca7f1785ad17713ae417a3ba87292630051a036c\": container with ID starting with ef30e8cfa3a3c4963cebc0ebca7f1785ad17713ae417a3ba87292630051a036c not found: ID does not exist" Oct 30 22:13:50.560345 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:50.560313 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-69e4b2-5845f89958-mzv58"] Oct 30 22:13:50.570702 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:50.570675 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-69e4b2-5845f89958-mzv58"] Oct 30 22:13:50.675492 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:50.675414 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-69e4b2-77c69fd5cf-5jr47"] Oct 30 22:13:50.683585 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:50.683564 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-69e4b2-77c69fd5cf-5jr47"] Oct 30 22:13:52.425155 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:52.425117 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08366fbc-97d1-41a0-959a-447d00b7ec88" path="/var/lib/kubelet/pods/08366fbc-97d1-41a0-959a-447d00b7ec88/volumes" Oct 30 22:13:52.425787 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:13:52.425652 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ecabb08-132c-4b22-b550-77dbff23555d" path="/var/lib/kubelet/pods/1ecabb08-132c-4b22-b550-77dbff23555d/volumes" Oct 30 22:14:00.143722 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:14:00.143686 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:14:30.143234 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:14:30.143192 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:14:38.529572 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:14:38.529541 5748 scope.go:122] "RemoveContainer" containerID="808a60df966cb339b36ba4e5c5c2769023cee8f02c58993d39bdda5c69dfecd4" Oct 30 22:14:38.538536 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:14:38.538511 5748 scope.go:122] "RemoveContainer" containerID="ce5ce3db0dfe5d7c7f3704a4e4bcd8927fc26c755b3f07307ee296112800f1b8" Oct 30 22:14:48.046515 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:14:48.046422 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d84e8fa-f8zg8"] Oct 30 22:14:48.096861 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:14:48.096820 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d84e8fa-f8zg8"] Oct 30 22:14:48.414786 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:14:48.414751 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0ac6c07-ace0-48dd-9922-5497406533e5" path="/var/lib/kubelet/pods/d0ac6c07-ace0-48dd-9922-5497406533e5/volumes" Oct 30 22:15:00.144781 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:15:00.144744 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:15:01.049696 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:15:01.049663 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5f78b46-w24k6"] Oct 30 22:15:01.100031 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:15:01.099998 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5f78b46-w24k6"] Oct 30 22:15:02.415409 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:15:02.415374 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd995578-e1ab-4647-8092-8ddf357b2060" path="/var/lib/kubelet/pods/dd995578-e1ab-4647-8092-8ddf357b2060/volumes" Oct 30 22:15:15.048374 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:15:15.048344 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92a1c4d5-c5crj"] Oct 30 22:15:15.098882 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:15:15.098851 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92a1c4d5-c5crj"] Oct 30 22:15:15.100313 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:15:15.100291 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29f06004-7ljm6"] Oct 30 22:15:15.101600 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:15:15.101578 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29f06004-7ljm6"] Oct 30 22:15:16.415574 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:15:16.415540 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88d13e16-48cd-4cbc-8569-c2b46e06f555" path="/var/lib/kubelet/pods/88d13e16-48cd-4cbc-8569-c2b46e06f555/volumes" Oct 30 22:15:16.416006 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:15:16.415959 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7e9d383-7fb2-4b0f-ad14-708eb31fb828" path="/var/lib/kubelet/pods/c7e9d383-7fb2-4b0f-ad14-708eb31fb828/volumes" Oct 30 22:15:26.047689 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:15:26.047654 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96d4b810-n9xn9"] Oct 30 22:15:26.098277 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:15:26.098239 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44020668-q6n64"] Oct 30 22:15:26.099591 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:15:26.099571 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96d4b810-n9xn9"] Oct 30 22:15:26.102225 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:15:26.102206 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44020668-q6n64"] Oct 30 22:15:26.414995 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:15:26.414960 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02d5d75d-0e70-43df-9218-ecbb91ea5a59" path="/var/lib/kubelet/pods/02d5d75d-0e70-43df-9218-ecbb91ea5a59/volumes" Oct 30 22:15:26.415383 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:15:26.415370 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40bb68cf-74bf-4b69-bc52-7df542f17d48" path="/var/lib/kubelet/pods/40bb68cf-74bf-4b69-bc52-7df542f17d48/volumes" Oct 30 22:15:30.144169 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:15:30.144133 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:15:31.052204 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:15:31.052171 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2566e01-flg2f"] Oct 30 22:15:31.102855 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:15:31.102824 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4aa6bbe1-zgmlk"] Oct 30 22:15:31.104123 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:15:31.104111 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2566e01-flg2f"] Oct 30 22:15:31.105249 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:15:31.105229 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4aa6bbe1-zgmlk"] Oct 30 22:15:32.415962 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:15:32.415923 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="123c12b7-4ff3-435a-8a6d-593218afbd35" path="/var/lib/kubelet/pods/123c12b7-4ff3-435a-8a6d-593218afbd35/volumes" Oct 30 22:15:32.416492 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:15:32.416359 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bfbddd0-67de-45c5-bb50-7f809c330b3e" path="/var/lib/kubelet/pods/5bfbddd0-67de-45c5-bb50-7f809c330b3e/volumes" Oct 30 22:15:38.570393 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:15:38.570361 5748 scope.go:122] "RemoveContainer" containerID="105101e191f029986dda1afe729cc343ece8ba67342d68c4466a046aa01f59b0" Oct 30 22:15:38.579412 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:15:38.579393 5748 scope.go:122] "RemoveContainer" containerID="f479376610f0c8c94fda5acb5adb19cd6c01e36a17dc0a152766d835127c5e63" Oct 30 22:15:56.047087 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:15:56.047046 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ce82c78-f9wrj"] Oct 30 22:15:56.097507 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:15:56.097474 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ce82c78-f9wrj"] Oct 30 22:15:56.415991 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:15:56.415958 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89a64fcf-55f5-4d7f-8a79-f97d61b582d1" path="/var/lib/kubelet/pods/89a64fcf-55f5-4d7f-8a79-f97d61b582d1/volumes" Oct 30 22:16:00.143945 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:16:00.143885 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:16:30.143944 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:16:30.143883 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:16:32.046986 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:16:32.046951 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4307febc-nxghv"] Oct 30 22:16:32.097284 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:16:32.097251 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4307febc-nxghv"] Oct 30 22:16:32.415016 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:16:32.414987 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f77651a-b47a-46f8-9d33-a6baa49bc9db" path="/var/lib/kubelet/pods/4f77651a-b47a-46f8-9d33-a6baa49bc9db/volumes" Oct 30 22:16:38.605086 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:16:38.605047 5748 scope.go:122] "RemoveContainer" containerID="564f8c162f64c5d4a00c67df0bbfc9cd856ceac08984c3b95facf4cd3536d865" Oct 30 22:16:38.613681 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:16:38.613654 5748 scope.go:122] "RemoveContainer" containerID="b3bfa28688f5dd4c624e37c066392f56f4ca0994fc7b9452ac836bbc2339901d" Oct 30 22:16:38.625308 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:16:38.625287 5748 scope.go:122] "RemoveContainer" containerID="8adcc008e518884e05136e75ddba09f51ff8191b5282f65bbd1dc02a30b4e7fa" Oct 30 22:16:38.633080 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:16:38.633061 5748 scope.go:122] "RemoveContainer" containerID="f50b657fa217395911ec06400f51900f5168eff0f1aa54ad1d32969c12b09b12" Oct 30 22:16:38.641179 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:16:38.641161 5748 scope.go:122] "RemoveContainer" containerID="e7914ef9e59bed0d1770f9497c787a59e602c3fbe1eb71a98f517a76d7ef138f" Oct 30 22:16:38.648539 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:16:38.648522 5748 scope.go:122] "RemoveContainer" containerID="5f31883435d611abedfa0b34d92224cb8c31d0f4d35c2d431cc5a293face6664" Oct 30 22:16:38.655829 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:16:38.655808 5748 scope.go:122] "RemoveContainer" containerID="ae7ab12574a9fb859e8a646d6befd78833522f3036badc7ba93e29f790c50ded" Oct 30 22:16:38.663097 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:16:38.663078 5748 scope.go:122] "RemoveContainer" containerID="209c37444bc4a54e54664e8607ac5e0303ea084baed33f0fdfe20787587b1351" Oct 30 22:16:38.670151 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:16:38.670136 5748 scope.go:122] "RemoveContainer" containerID="964d2125e4b150d0b8848f491c1b0150900db97260e035495c920bf353ca089a" Oct 30 22:16:38.677446 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:16:38.677429 5748 scope.go:122] "RemoveContainer" containerID="480bff8d12d6fcff995cebae2113f89a9561da1c4fad4826e2ae6343cfa79fae" Oct 30 22:16:38.684565 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:16:38.684550 5748 scope.go:122] "RemoveContainer" containerID="7db59b582526e2ec6fa0ca9353423290bdc117d776073e06523dadb4ce3e6d1f" Oct 30 22:16:58.047490 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:16:58.047452 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a0395a5-mjcrk"] Oct 30 22:16:58.097852 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:16:58.097812 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a0395a5-mjcrk"] Oct 30 22:16:58.415013 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:16:58.414980 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe298e2d-f999-4757-afa1-efb5928ca9ca" path="/var/lib/kubelet/pods/fe298e2d-f999-4757-afa1-efb5928ca9ca/volumes" Oct 30 22:16:59.045454 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:16:59.045420 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de309da5-rnj2v"] Oct 30 22:16:59.095932 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:16:59.095882 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de309da5-rnj2v"] Oct 30 22:17:00.194717 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:17:00.194684 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:17:00.452024 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:17:00.443017 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37f5380a-6450-448b-9f2e-39c0464dce94" path="/var/lib/kubelet/pods/37f5380a-6450-448b-9f2e-39c0464dce94/volumes" Oct 30 22:17:30.143376 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:17:30.143336 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:17:38.738340 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:17:38.738308 5748 scope.go:122] "RemoveContainer" containerID="c399ddfb9d2ed1570bd8cce9b78676e8deeb686d751660e29c561fd19b421336" Oct 30 22:17:38.747164 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:17:38.747139 5748 scope.go:122] "RemoveContainer" containerID="c8ad86110fc66df2ee6f01493035a478ad7d68a82d6910414ca9dd893bbfe5cc" Oct 30 22:17:38.754951 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:17:38.754932 5748 scope.go:122] "RemoveContainer" containerID="edcb20a0396a3fcf21de0c3fa4e274b991bb69c5fa69fd3dc9a1d73e71410380" Oct 30 22:18:00.144379 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:18:00.144341 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:18:30.143854 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:18:30.143809 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:18:31.047515 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:18:31.047445 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91bf7838-9vvk4"] Oct 30 22:18:31.098267 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:18:31.098232 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91bf7838-9vvk4"] Oct 30 22:18:32.417209 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:18:32.417175 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92547c4c-d9ff-4b5b-a8f5-dff883c46899" path="/var/lib/kubelet/pods/92547c4c-d9ff-4b5b-a8f5-dff883c46899/volumes" Oct 30 22:18:36.944826 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:18:36.944797 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j4v8r_3df036ff-1c00-4ce2-9e86-8acbeedf6c82/ovn-acl-logging/0.log" Oct 30 22:18:36.958070 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:18:36.958049 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j4v8r_3df036ff-1c00-4ce2-9e86-8acbeedf6c82/ovn-acl-logging/0.log" Oct 30 22:18:38.780575 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:18:38.780542 5748 scope.go:122] "RemoveContainer" containerID="bfd635cfd7766eb7c7948860f6de1d43f2f15304668811517899ec34cf52610a" Oct 30 22:18:38.789141 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:18:38.789122 5748 scope.go:122] "RemoveContainer" containerID="1a351ec478d1baee6660bc4a9fb1f789c5e5a3969cba661bb9b614fa7d260d7d" Oct 30 22:18:38.796622 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:18:38.796605 5748 scope.go:122] "RemoveContainer" containerID="c7b89a2ad4c2e45cbf94b550ba7ba26da8dbdfe4b008d936e657a16e2540c4f7" Oct 30 22:18:38.803855 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:18:38.803833 5748 scope.go:122] "RemoveContainer" containerID="6fec63d1e512a80b49f4dd46c6959f7e2a3f3e56483ec045d45b1f36486e75f8" Oct 30 22:19:00.145078 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:19:00.145033 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:19:30.144509 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:19:30.144467 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:19:38.835273 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:19:38.835231 5748 scope.go:122] "RemoveContainer" containerID="aa018b4ba5c912de99ce7983bcc47a2c41d3af64e0706d0f7e94548d08f63786" Oct 30 22:19:38.845982 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:19:38.845954 5748 scope.go:122] "RemoveContainer" containerID="bbb54838fdde62a75e2bf442b14182076781a635ec8eaa2a73ae1be83deb4144" Oct 30 22:19:38.855173 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:19:38.855146 5748 scope.go:122] "RemoveContainer" containerID="1457707154a239c34dacc2beb9c6edcbc33cc64681cb708d8be17c3eb91a10f5" Oct 30 22:19:40.173919 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:19:40.172997 5748 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-t7djv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:19:40.173919 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:19:40.173075 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t7djv" podUID="818838b2-f54f-4591-b855-73014d18a06e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:20:00.145011 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:20:00.144973 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:20:30.144401 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:20:30.144362 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:21:00.143649 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:21:00.143611 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:21:30.151108 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:21:30.151057 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:22:00.144284 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:22:00.144246 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:22:03.052922 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:22:03.051984 5748 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-948nx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:22:03.052922 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:22:03.052067 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-948nx" podUID="f411f535-b1bc-408b-afb9-3299fc9d5bd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:22:13.312476 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:22:13.312423 5748 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-98fnd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:22:13.313116 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:22:13.313075 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-98fnd" podUID="d1c1a2b1-f312-47b1-b437-2d8f1760eea0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:22:30.186392 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:22:30.186353 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:23:00.144822 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:23:00.144785 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:23:30.146744 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:23:30.146699 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:23:36.974551 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:23:36.974520 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j4v8r_3df036ff-1c00-4ce2-9e86-8acbeedf6c82/ovn-acl-logging/0.log" Oct 30 22:23:36.988179 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:23:36.988156 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j4v8r_3df036ff-1c00-4ce2-9e86-8acbeedf6c82/ovn-acl-logging/0.log" Oct 30 22:24:00.145220 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:24:00.145179 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:24:30.146439 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:24:30.146391 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:25:00.143398 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:25:00.143361 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:25:30.144218 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:25:30.144179 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:26:00.144235 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:26:00.144195 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:26:30.144131 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:26:30.144091 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:27:00.143949 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:27:00.143857 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:27:10.182089 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:27:10.182045 5748 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-t7djv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:27:10.182605 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:27:10.182133 5748 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t7djv" podUID="818838b2-f54f-4591-b855-73014d18a06e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:27:30.143951 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:27:30.143911 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:28:00.143597 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:28:00.143562 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:28:30.143953 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:28:30.143915 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:28:37.002644 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:28:37.002611 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j4v8r_3df036ff-1c00-4ce2-9e86-8acbeedf6c82/ovn-acl-logging/0.log" Oct 30 22:28:37.026252 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:28:37.026226 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j4v8r_3df036ff-1c00-4ce2-9e86-8acbeedf6c82/ovn-acl-logging/0.log" Oct 30 22:29:00.144851 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:29:00.144814 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:29:30.147572 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:29:30.147535 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:30:00.144006 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:30:00.143921 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:30:00.170293 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:30:00.170265 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-b24ff0-2988999k8rxl"] Oct 30 22:30:00.170659 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:30:00.170646 5748 cpu_manager.go:414] "containerMap: removing container" podUID="3ed43145-33bd-4c11-bb29-dc852511f6d2" containerName="main" Oct 30 22:30:00.170707 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:30:00.170660 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ed43145-33bd-4c11-bb29-dc852511f6d2" containerName="main" Oct 30 22:30:00.170707 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:30:00.170671 5748 cpu_manager.go:414] "containerMap: removing container" podUID="08366fbc-97d1-41a0-959a-447d00b7ec88" containerName="payload-pre-processing" Oct 30 22:30:00.170707 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:30:00.170677 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08366fbc-97d1-41a0-959a-447d00b7ec88" containerName="payload-pre-processing" Oct 30 22:30:00.170707 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:30:00.170685 5748 cpu_manager.go:414] "containerMap: removing container" podUID="698fc0f2-64ce-4e7e-b482-8ebd9f33f84d" containerName="main" Oct 30 22:30:00.170707 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:30:00.170690 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="698fc0f2-64ce-4e7e-b482-8ebd9f33f84d" containerName="main" Oct 30 22:30:00.170707 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:30:00.170703 5748 cpu_manager.go:414] "containerMap: removing container" podUID="1ecabb08-132c-4b22-b550-77dbff23555d" containerName="payload-processing" Oct 30 22:30:00.170707 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:30:00.170708 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ecabb08-132c-4b22-b550-77dbff23555d" containerName="payload-processing" Oct 30 22:30:00.170975 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:30:00.170729 5748 cpu_manager.go:414] "containerMap: removing container" podUID="92547c4c-d9ff-4b5b-a8f5-dff883c46899" containerName="revoke-keys" Oct 30 22:30:00.170975 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:30:00.170734 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92547c4c-d9ff-4b5b-a8f5-dff883c46899" containerName="revoke-keys" Oct 30 22:30:00.170975 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:30:00.170793 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ed43145-33bd-4c11-bb29-dc852511f6d2" containerName="main" Oct 30 22:30:00.170975 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:30:00.170801 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="08366fbc-97d1-41a0-959a-447d00b7ec88" containerName="payload-pre-processing" Oct 30 22:30:00.170975 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:30:00.170807 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="698fc0f2-64ce-4e7e-b482-8ebd9f33f84d" containerName="main" Oct 30 22:30:00.170975 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:30:00.170814 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ecabb08-132c-4b22-b550-77dbff23555d" containerName="payload-processing" Oct 30 22:30:00.170975 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:30:00.170823 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="92547c4c-d9ff-4b5b-a8f5-dff883c46899" containerName="revoke-keys" Oct 30 22:30:00.173886 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:30:00.173872 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-b24ff0-2988999k8rxl" Oct 30 22:30:00.229755 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:30:00.229724 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:30:00.277694 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:30:00.277663 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:30:00.277848 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:30:00.277736 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 22:30:00.279860 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:30:00.279824 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-b24ff0-2988999k8rxl"] Oct 30 22:30:00.289792 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:30:00.289762 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4g6l\" (UniqueName: \"kubernetes.io/projected/cd7143f9-5126-4c90-b570-7c17c282a5cd-kube-api-access-w4g6l\") pod \"maas-api-key-cleanup-e2e-worker-single-main-b24ff0-2988999k8rxl\" (UID: \"cd7143f9-5126-4c90-b570-7c17c282a5cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-b24ff0-2988999k8rxl" Oct 30 22:30:00.289996 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:30:00.289971 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd7143f9-5126-4c90-b570-7c17c282a5cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-b24ff0-2988999k8rxl\" (UID: \"cd7143f9-5126-4c90-b570-7c17c282a5cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-b24ff0-2988999k8rxl" Oct 30 22:30:00.390865 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:30:00.390833 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4g6l\" (UniqueName: \"kubernetes.io/projected/cd7143f9-5126-4c90-b570-7c17c282a5cd-kube-api-access-w4g6l\") pod \"maas-api-key-cleanup-e2e-worker-single-main-b24ff0-2988999k8rxl\" (UID: \"cd7143f9-5126-4c90-b570-7c17c282a5cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-b24ff0-2988999k8rxl" Oct 30 22:30:00.391062 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:30:00.390969 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd7143f9-5126-4c90-b570-7c17c282a5cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-b24ff0-2988999k8rxl\" (UID: \"cd7143f9-5126-4c90-b570-7c17c282a5cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-b24ff0-2988999k8rxl" Oct 30 22:30:00.391575 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:30:00.391548 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd7143f9-5126-4c90-b570-7c17c282a5cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-b24ff0-2988999k8rxl\" (UID: \"cd7143f9-5126-4c90-b570-7c17c282a5cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-b24ff0-2988999k8rxl" Oct 30 22:30:00.452761 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:30:00.452697 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4g6l\" (UniqueName: \"kubernetes.io/projected/cd7143f9-5126-4c90-b570-7c17c282a5cd-kube-api-access-w4g6l\") pod \"maas-api-key-cleanup-e2e-worker-single-main-b24ff0-2988999k8rxl\" (UID: \"cd7143f9-5126-4c90-b570-7c17c282a5cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-b24ff0-2988999k8rxl" Oct 30 22:30:00.486524 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:30:00.486500 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-b24ff0-2988999k8rxl" Oct 30 22:30:00.859909 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:30:00.859871 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-b24ff0-2988999k8rxl"] Oct 30 22:30:00.862745 ip-10-0-143-61 kubenswrapper[5748]: W1030 22:30:00.862708 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd7143f9_5126_4c90_b570_7c17c282a5cd.slice/crio-198252393f76124af8e3ddd73f50e4fc27e4bca890e7b570fbdcca9d26013104 WatchSource:0}: Error finding container 198252393f76124af8e3ddd73f50e4fc27e4bca890e7b570fbdcca9d26013104: Status 404 returned error can't find the container with id 198252393f76124af8e3ddd73f50e4fc27e4bca890e7b570fbdcca9d26013104 Oct 30 22:30:01.402097 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:30:01.402063 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-b24ff0-2988999k8rxl" event={"ID":"cd7143f9-5126-4c90-b570-7c17c282a5cd","Type":"ContainerStarted","Data":"a35e8298128bb2eed8d3f2461aebc61d39d6922792b13d4ff49b692832663ba6"} Oct 30 22:30:01.402515 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:30:01.402113 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-b24ff0-2988999k8rxl" event={"ID":"cd7143f9-5126-4c90-b570-7c17c282a5cd","Type":"ContainerStarted","Data":"198252393f76124af8e3ddd73f50e4fc27e4bca890e7b570fbdcca9d26013104"} Oct 30 22:30:01.518728 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:30:01.518670 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-b24ff0-2988999k8rxl" podStartSLOduration=1.518655839 podStartE2EDuration="1.518655839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:01.518095008 +0000 UTC m=+3985.260615956" watchObservedRunningTime="2026-10-30 22:30:01.518655839 +0000 UTC m=+3985.261176774" Oct 30 22:30:02.407571 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:30:02.407531 5748 generic.go:356] "Generic (PLEG): container finished" podID="cd7143f9-5126-4c90-b570-7c17c282a5cd" containerID="a35e8298128bb2eed8d3f2461aebc61d39d6922792b13d4ff49b692832663ba6" exitCode=60 Oct 30 22:30:02.408091 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:30:02.407620 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-b24ff0-2988999k8rxl" event={"ID":"cd7143f9-5126-4c90-b570-7c17c282a5cd","Type":"ContainerDied","Data":"a35e8298128bb2eed8d3f2461aebc61d39d6922792b13d4ff49b692832663ba6"} Oct 30 22:30:02.408091 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:30:02.407871 5748 scope.go:122] "RemoveContainer" containerID="a35e8298128bb2eed8d3f2461aebc61d39d6922792b13d4ff49b692832663ba6" Oct 30 22:30:03.413844 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:30:03.413812 5748 generic.go:356] "Generic (PLEG): container finished" podID="cd7143f9-5126-4c90-b570-7c17c282a5cd" containerID="487608f412e63260a5c99ac712c7ca566663fd4b5d71bc5cc2d4ed7ff994e54a" exitCode=60 Oct 30 22:30:03.414316 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:30:03.413908 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-b24ff0-2988999k8rxl" event={"ID":"cd7143f9-5126-4c90-b570-7c17c282a5cd","Type":"ContainerDied","Data":"487608f412e63260a5c99ac712c7ca566663fd4b5d71bc5cc2d4ed7ff994e54a"} Oct 30 22:30:03.414316 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:30:03.413976 5748 scope.go:122] "RemoveContainer" containerID="a35e8298128bb2eed8d3f2461aebc61d39d6922792b13d4ff49b692832663ba6" Oct 30 22:30:03.414316 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:30:03.414132 5748 scope.go:122] "RemoveContainer" containerID="487608f412e63260a5c99ac712c7ca566663fd4b5d71bc5cc2d4ed7ff994e54a" Oct 30 22:30:03.414316 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:30:03.414307 5748 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-b24ff0-2988999k8rxl_odh-ai-gateway-infra(cd7143f9-5126-4c90-b570-7c17c282a5cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-b24ff0-2988999k8rxl" podUID="cd7143f9-5126-4c90-b570-7c17c282a5cd" Oct 30 22:30:04.419507 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:30:04.419479 5748 scope.go:122] "RemoveContainer" containerID="487608f412e63260a5c99ac712c7ca566663fd4b5d71bc5cc2d4ed7ff994e54a" Oct 30 22:30:04.419874 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:30:04.419682 5748 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-b24ff0-2988999k8rxl_odh-ai-gateway-infra(cd7143f9-5126-4c90-b570-7c17c282a5cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-b24ff0-2988999k8rxl" podUID="cd7143f9-5126-4c90-b570-7c17c282a5cd" Oct 30 22:30:13.282280 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:30:13.282246 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-b24ff0-2988999k8rxl"] Oct 30 22:30:13.422116 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:30:13.422086 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-b24ff0-2988999k8rxl" Oct 30 22:30:13.457091 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:30:13.457064 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-b24ff0-2988999k8rxl" Oct 30 22:30:13.457235 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:30:13.457102 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-b24ff0-2988999k8rxl" event={"ID":"cd7143f9-5126-4c90-b570-7c17c282a5cd","Type":"ContainerDied","Data":"198252393f76124af8e3ddd73f50e4fc27e4bca890e7b570fbdcca9d26013104"} Oct 30 22:30:13.457235 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:30:13.457161 5748 scope.go:122] "RemoveContainer" containerID="487608f412e63260a5c99ac712c7ca566663fd4b5d71bc5cc2d4ed7ff994e54a" Oct 30 22:30:13.512532 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:30:13.512503 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd7143f9-5126-4c90-b570-7c17c282a5cd-kube-api-access-w4g6l\" (UniqueName: \"kubernetes.io/projected/cd7143f9-5126-4c90-b570-7c17c282a5cd-kube-api-access-w4g6l\") pod \"cd7143f9-5126-4c90-b570-7c17c282a5cd\" (UID: \"cd7143f9-5126-4c90-b570-7c17c282a5cd\") " Oct 30 22:30:13.512685 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:30:13.512569 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd7143f9-5126-4c90-b570-7c17c282a5cd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd7143f9-5126-4c90-b570-7c17c282a5cd-maas-api-service-ca\") pod \"cd7143f9-5126-4c90-b570-7c17c282a5cd\" (UID: \"cd7143f9-5126-4c90-b570-7c17c282a5cd\") " Oct 30 22:30:13.512949 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:30:13.512891 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd7143f9-5126-4c90-b570-7c17c282a5cd-maas-api-service-ca" pod "cd7143f9-5126-4c90-b570-7c17c282a5cd" (UID: "cd7143f9-5126-4c90-b570-7c17c282a5cd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:13.514598 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:30:13.514574 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd7143f9-5126-4c90-b570-7c17c282a5cd-kube-api-access-w4g6l" pod "cd7143f9-5126-4c90-b570-7c17c282a5cd" (UID: "cd7143f9-5126-4c90-b570-7c17c282a5cd"). InnerVolumeSpecName "kube-api-access-w4g6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:13.613783 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:30:13.613702 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4g6l\" (UniqueName: \"kubernetes.io/projected/cd7143f9-5126-4c90-b570-7c17c282a5cd-kube-api-access-w4g6l\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:13.613783 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:30:13.613735 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd7143f9-5126-4c90-b570-7c17c282a5cd-maas-api-service-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:13.897492 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:30:13.897450 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-b24ff0-2988999k8rxl"] Oct 30 22:30:13.907161 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:30:13.907135 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-b24ff0-2988999k8rxl"] Oct 30 22:30:14.415697 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:30:14.415662 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd7143f9-5126-4c90-b570-7c17c282a5cd" path="/var/lib/kubelet/pods/cd7143f9-5126-4c90-b570-7c17c282a5cd/volumes" Oct 30 22:30:30.143240 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:30:30.143202 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:31:00.143808 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:31:00.143772 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:31:30.144291 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:31:30.144249 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:32:00.192828 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:32:00.192785 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:32:30.144866 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:32:30.144827 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:33:00.145112 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:33:00.145032 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:33:30.144229 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:33:30.144183 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:33:37.031253 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:33:37.031217 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j4v8r_3df036ff-1c00-4ce2-9e86-8acbeedf6c82/ovn-acl-logging/0.log" Oct 30 22:33:37.055990 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:33:37.055970 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j4v8r_3df036ff-1c00-4ce2-9e86-8acbeedf6c82/ovn-acl-logging/0.log" Oct 30 22:33:44.467075 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:33:44.467034 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9871e8fb-rhw26"] Oct 30 22:33:44.467482 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:33:44.467420 5748 cpu_manager.go:414] "containerMap: removing container" podUID="cd7143f9-5126-4c90-b570-7c17c282a5cd" containerName="cleanup" Oct 30 22:33:44.467482 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:33:44.467431 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd7143f9-5126-4c90-b570-7c17c282a5cd" containerName="cleanup" Oct 30 22:33:44.467558 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:33:44.467515 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd7143f9-5126-4c90-b570-7c17c282a5cd" containerName="cleanup" Oct 30 22:33:44.467558 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:33:44.467525 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd7143f9-5126-4c90-b570-7c17c282a5cd" containerName="cleanup" Oct 30 22:33:44.470663 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:33:44.470645 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9871e8fb-rhw26" Oct 30 22:33:44.503799 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:33:44.503764 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rnq4\" (UniqueName: \"kubernetes.io/projected/a2f1a195-4759-4e05-8809-ddd897c2a7b7-kube-api-access-6rnq4\") pod \"maas-api-revoke-keys-e2e-ait-9871e8fb-rhw26\" (UID: \"a2f1a195-4759-4e05-8809-ddd897c2a7b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9871e8fb-rhw26" Oct 30 22:33:44.503966 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:33:44.503826 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2f1a195-4759-4e05-8809-ddd897c2a7b7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-9871e8fb-rhw26\" (UID: \"a2f1a195-4759-4e05-8809-ddd897c2a7b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9871e8fb-rhw26" Oct 30 22:33:44.503966 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:33:44.503891 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2f1a195-4759-4e05-8809-ddd897c2a7b7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-9871e8fb-rhw26\" (UID: \"a2f1a195-4759-4e05-8809-ddd897c2a7b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9871e8fb-rhw26" Oct 30 22:33:44.523523 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:33:44.523499 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:33:44.573882 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:33:44.573852 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:33:44.574055 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:33:44.573919 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:33:44.574055 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:33:44.573994 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mdpxk\"" Oct 30 22:33:44.575407 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:33:44.575385 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9871e8fb-rhw26"] Oct 30 22:33:44.604777 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:33:44.604751 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2f1a195-4759-4e05-8809-ddd897c2a7b7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-9871e8fb-rhw26\" (UID: \"a2f1a195-4759-4e05-8809-ddd897c2a7b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9871e8fb-rhw26" Oct 30 22:33:44.604891 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:33:44.604786 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2f1a195-4759-4e05-8809-ddd897c2a7b7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-9871e8fb-rhw26\" (UID: \"a2f1a195-4759-4e05-8809-ddd897c2a7b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9871e8fb-rhw26" Oct 30 22:33:44.604891 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:33:44.604846 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rnq4\" (UniqueName: \"kubernetes.io/projected/a2f1a195-4759-4e05-8809-ddd897c2a7b7-kube-api-access-6rnq4\") pod \"maas-api-revoke-keys-e2e-ait-9871e8fb-rhw26\" (UID: \"a2f1a195-4759-4e05-8809-ddd897c2a7b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9871e8fb-rhw26" Oct 30 22:33:44.605395 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:33:44.605375 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2f1a195-4759-4e05-8809-ddd897c2a7b7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-9871e8fb-rhw26\" (UID: \"a2f1a195-4759-4e05-8809-ddd897c2a7b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9871e8fb-rhw26" Oct 30 22:33:44.607155 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:33:44.607135 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2f1a195-4759-4e05-8809-ddd897c2a7b7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-9871e8fb-rhw26\" (UID: \"a2f1a195-4759-4e05-8809-ddd897c2a7b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9871e8fb-rhw26" Oct 30 22:33:44.666360 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:33:44.666331 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rnq4\" (UniqueName: \"kubernetes.io/projected/a2f1a195-4759-4e05-8809-ddd897c2a7b7-kube-api-access-6rnq4\") pod \"maas-api-revoke-keys-e2e-ait-9871e8fb-rhw26\" (UID: \"a2f1a195-4759-4e05-8809-ddd897c2a7b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9871e8fb-rhw26" Oct 30 22:33:44.784489 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:33:44.784430 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9871e8fb-rhw26" Oct 30 22:33:45.157763 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:33:45.157617 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9871e8fb-rhw26"] Oct 30 22:33:45.160579 ip-10-0-143-61 kubenswrapper[5748]: W1030 22:33:45.160545 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2f1a195_4759_4e05_8809_ddd897c2a7b7.slice/crio-51ea8acc58104a6d7ae155f2c8bb5ab86524165764b73548a9f4bbbd8ed1d088 WatchSource:0}: Error finding container 51ea8acc58104a6d7ae155f2c8bb5ab86524165764b73548a9f4bbbd8ed1d088: Status 404 returned error can't find the container with id 51ea8acc58104a6d7ae155f2c8bb5ab86524165764b73548a9f4bbbd8ed1d088 Oct 30 22:33:45.498559 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:33:45.498527 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9871e8fb-rhw26" event={"ID":"a2f1a195-4759-4e05-8809-ddd897c2a7b7","Type":"ContainerStarted","Data":"8bc066a35311bff869c7fc1722ed7db5e9513c241e70891d696f50d95dfff2ca"} Oct 30 22:33:45.499012 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:33:45.498587 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9871e8fb-rhw26" event={"ID":"a2f1a195-4759-4e05-8809-ddd897c2a7b7","Type":"ContainerStarted","Data":"51ea8acc58104a6d7ae155f2c8bb5ab86524165764b73548a9f4bbbd8ed1d088"} Oct 30 22:33:45.616224 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:33:45.616167 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9871e8fb-rhw26" podStartSLOduration=1.616152512 podStartE2EDuration="1.616152512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:33:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:33:45.615910062 +0000 UTC m=+4209.358430993" watchObservedRunningTime="2026-10-30 22:33:45.616152512 +0000 UTC m=+4209.358673447" Oct 30 22:33:46.504545 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:33:46.504510 5748 generic.go:356] "Generic (PLEG): container finished" podID="a2f1a195-4759-4e05-8809-ddd897c2a7b7" containerID="8bc066a35311bff869c7fc1722ed7db5e9513c241e70891d696f50d95dfff2ca" exitCode=0 Oct 30 22:33:46.504942 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:33:46.504571 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9871e8fb-rhw26" event={"ID":"a2f1a195-4759-4e05-8809-ddd897c2a7b7","Type":"ContainerDied","Data":"8bc066a35311bff869c7fc1722ed7db5e9513c241e70891d696f50d95dfff2ca"} Oct 30 22:33:47.640448 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:33:47.640415 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9871e8fb-rhw26" Oct 30 22:33:47.735769 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:33:47.735736 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2f1a195-4759-4e05-8809-ddd897c2a7b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2f1a195-4759-4e05-8809-ddd897c2a7b7-maas-api-openshift-ca\") pod \"a2f1a195-4759-4e05-8809-ddd897c2a7b7\" (UID: \"a2f1a195-4759-4e05-8809-ddd897c2a7b7\") " Oct 30 22:33:47.735965 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:33:47.735793 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2f1a195-4759-4e05-8809-ddd897c2a7b7-kube-api-access-6rnq4\" (UniqueName: \"kubernetes.io/projected/a2f1a195-4759-4e05-8809-ddd897c2a7b7-kube-api-access-6rnq4\") pod \"a2f1a195-4759-4e05-8809-ddd897c2a7b7\" (UID: \"a2f1a195-4759-4e05-8809-ddd897c2a7b7\") " Oct 30 22:33:47.735965 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:33:47.735833 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2f1a195-4759-4e05-8809-ddd897c2a7b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2f1a195-4759-4e05-8809-ddd897c2a7b7-maas-api-xks-ca\") pod \"a2f1a195-4759-4e05-8809-ddd897c2a7b7\" (UID: \"a2f1a195-4759-4e05-8809-ddd897c2a7b7\") " Oct 30 22:33:47.736191 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:33:47.736113 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2f1a195-4759-4e05-8809-ddd897c2a7b7-maas-api-openshift-ca" pod "a2f1a195-4759-4e05-8809-ddd897c2a7b7" (UID: "a2f1a195-4759-4e05-8809-ddd897c2a7b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:33:47.737949 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:33:47.737917 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2f1a195-4759-4e05-8809-ddd897c2a7b7-maas-api-xks-ca" pod "a2f1a195-4759-4e05-8809-ddd897c2a7b7" (UID: "a2f1a195-4759-4e05-8809-ddd897c2a7b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:33:47.738077 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:33:47.738019 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2f1a195-4759-4e05-8809-ddd897c2a7b7-kube-api-access-6rnq4" pod "a2f1a195-4759-4e05-8809-ddd897c2a7b7" (UID: "a2f1a195-4759-4e05-8809-ddd897c2a7b7"). InnerVolumeSpecName "kube-api-access-6rnq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:33:47.837113 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:33:47.837046 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2f1a195-4759-4e05-8809-ddd897c2a7b7-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:33:47.837113 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:33:47.837073 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rnq4\" (UniqueName: \"kubernetes.io/projected/a2f1a195-4759-4e05-8809-ddd897c2a7b7-kube-api-access-6rnq4\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:33:47.837113 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:33:47.837083 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2f1a195-4759-4e05-8809-ddd897c2a7b7-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:33:48.515954 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:33:48.515924 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9871e8fb-rhw26" Oct 30 22:33:48.516130 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:33:48.515923 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9871e8fb-rhw26" event={"ID":"a2f1a195-4759-4e05-8809-ddd897c2a7b7","Type":"ContainerDied","Data":"51ea8acc58104a6d7ae155f2c8bb5ab86524165764b73548a9f4bbbd8ed1d088"} Oct 30 22:33:48.516130 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:33:48.516039 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51ea8acc58104a6d7ae155f2c8bb5ab86524165764b73548a9f4bbbd8ed1d088" Oct 30 22:34:00.147069 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:34:00.147030 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:34:04.838260 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:34:04.838157 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9871e8fb-rhw26"] Oct 30 22:34:04.887477 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:34:04.887444 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9871e8fb-rhw26"] Oct 30 22:34:06.415205 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:34:06.415171 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2f1a195-4759-4e05-8809-ddd897c2a7b7" path="/var/lib/kubelet/pods/a2f1a195-4759-4e05-8809-ddd897c2a7b7/volumes" Oct 30 22:34:30.144008 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:34:30.143968 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:35:00.144157 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:35:00.144118 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:35:30.143542 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:35:30.143502 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:36:00.144577 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:36:00.144472 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:36:18.085306 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:36:18.085268 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-563990-bzql6"] Oct 30 22:36:18.085883 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:36:18.085860 5748 cpu_manager.go:414] "containerMap: removing container" podUID="cd7143f9-5126-4c90-b570-7c17c282a5cd" containerName="cleanup" Oct 30 22:36:18.086012 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:36:18.085886 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd7143f9-5126-4c90-b570-7c17c282a5cd" containerName="cleanup" Oct 30 22:36:18.086012 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:36:18.085914 5748 cpu_manager.go:414] "containerMap: removing container" podUID="a2f1a195-4759-4e05-8809-ddd897c2a7b7" containerName="revoke-keys" Oct 30 22:36:18.086012 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:36:18.085922 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2f1a195-4759-4e05-8809-ddd897c2a7b7" containerName="revoke-keys" Oct 30 22:36:18.086181 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:36:18.086021 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2f1a195-4759-4e05-8809-ddd897c2a7b7" containerName="revoke-keys" Oct 30 22:36:18.088957 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:36:18.088935 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-563990-bzql6" Oct 30 22:36:18.142019 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:36:18.141996 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:36:18.192224 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:36:18.192200 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:36:18.192357 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:36:18.192230 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:36:18.192357 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:36:18.192248 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mdpxk\"" Oct 30 22:36:18.193560 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:36:18.193541 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-563990-bzql6"] Oct 30 22:36:18.211948 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:36:18.211922 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eedcc391-69e5-4cc8-bbbc-47817715269e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-563990-bzql6\" (UID: \"eedcc391-69e5-4cc8-bbbc-47817715269e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-563990-bzql6" Oct 30 22:36:18.212074 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:36:18.211995 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eedcc391-69e5-4cc8-bbbc-47817715269e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-563990-bzql6\" (UID: \"eedcc391-69e5-4cc8-bbbc-47817715269e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-563990-bzql6" Oct 30 22:36:18.212136 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:36:18.212070 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6svj\" (UniqueName: \"kubernetes.io/projected/eedcc391-69e5-4cc8-bbbc-47817715269e-kube-api-access-v6svj\") pod \"maas-api-revoke-keys-e2e-authcfg-http-563990-bzql6\" (UID: \"eedcc391-69e5-4cc8-bbbc-47817715269e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-563990-bzql6" Oct 30 22:36:18.312589 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:36:18.312557 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eedcc391-69e5-4cc8-bbbc-47817715269e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-563990-bzql6\" (UID: \"eedcc391-69e5-4cc8-bbbc-47817715269e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-563990-bzql6" Oct 30 22:36:18.312751 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:36:18.312620 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6svj\" (UniqueName: \"kubernetes.io/projected/eedcc391-69e5-4cc8-bbbc-47817715269e-kube-api-access-v6svj\") pod \"maas-api-revoke-keys-e2e-authcfg-http-563990-bzql6\" (UID: \"eedcc391-69e5-4cc8-bbbc-47817715269e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-563990-bzql6" Oct 30 22:36:18.312751 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:36:18.312718 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eedcc391-69e5-4cc8-bbbc-47817715269e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-563990-bzql6\" (UID: \"eedcc391-69e5-4cc8-bbbc-47817715269e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-563990-bzql6" Oct 30 22:36:18.313405 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:36:18.313381 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eedcc391-69e5-4cc8-bbbc-47817715269e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-563990-bzql6\" (UID: \"eedcc391-69e5-4cc8-bbbc-47817715269e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-563990-bzql6" Oct 30 22:36:18.314966 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:36:18.314945 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eedcc391-69e5-4cc8-bbbc-47817715269e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-563990-bzql6\" (UID: \"eedcc391-69e5-4cc8-bbbc-47817715269e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-563990-bzql6" Oct 30 22:36:18.374372 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:36:18.374296 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6svj\" (UniqueName: \"kubernetes.io/projected/eedcc391-69e5-4cc8-bbbc-47817715269e-kube-api-access-v6svj\") pod \"maas-api-revoke-keys-e2e-authcfg-http-563990-bzql6\" (UID: \"eedcc391-69e5-4cc8-bbbc-47817715269e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-563990-bzql6" Oct 30 22:36:18.400848 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:36:18.400814 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-563990-bzql6" Oct 30 22:36:18.783767 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:36:18.783731 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-563990-bzql6"] Oct 30 22:36:18.787683 ip-10-0-143-61 kubenswrapper[5748]: W1030 22:36:18.787654 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeedcc391_69e5_4cc8_bbbc_47817715269e.slice/crio-db70df110df281c85fd5ecee16e54b5aa445ffb4f8d5a3d7e597a47775032d07 WatchSource:0}: Error finding container db70df110df281c85fd5ecee16e54b5aa445ffb4f8d5a3d7e597a47775032d07: Status 404 returned error can't find the container with id db70df110df281c85fd5ecee16e54b5aa445ffb4f8d5a3d7e597a47775032d07 Oct 30 22:36:19.195921 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:36:19.195866 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-563990-bzql6" event={"ID":"eedcc391-69e5-4cc8-bbbc-47817715269e","Type":"ContainerStarted","Data":"9fb67fc393b6118cef2908b8c98505510ec85f3051add790de229c2b2d1fdd27"} Oct 30 22:36:19.196306 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:36:19.195934 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-563990-bzql6" event={"ID":"eedcc391-69e5-4cc8-bbbc-47817715269e","Type":"ContainerStarted","Data":"db70df110df281c85fd5ecee16e54b5aa445ffb4f8d5a3d7e597a47775032d07"} Oct 30 22:36:19.335915 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:36:19.335848 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-563990-bzql6" podStartSLOduration=1.3358338060000001 podStartE2EDuration="1.335833806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:36:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:36:19.33532317 +0000 UTC m=+4363.077844108" watchObservedRunningTime="2026-10-30 22:36:19.335833806 +0000 UTC m=+4363.078354741" Oct 30 22:36:20.202438 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:36:20.202404 5748 generic.go:356] "Generic (PLEG): container finished" podID="eedcc391-69e5-4cc8-bbbc-47817715269e" containerID="9fb67fc393b6118cef2908b8c98505510ec85f3051add790de229c2b2d1fdd27" exitCode=0 Oct 30 22:36:20.202832 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:36:20.202499 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-563990-bzql6" event={"ID":"eedcc391-69e5-4cc8-bbbc-47817715269e","Type":"ContainerDied","Data":"9fb67fc393b6118cef2908b8c98505510ec85f3051add790de229c2b2d1fdd27"} Oct 30 22:36:21.334152 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:36:21.334126 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-563990-bzql6" Oct 30 22:36:21.443855 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:36:21.443827 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eedcc391-69e5-4cc8-bbbc-47817715269e-kube-api-access-v6svj\" (UniqueName: \"kubernetes.io/projected/eedcc391-69e5-4cc8-bbbc-47817715269e-kube-api-access-v6svj\") pod \"eedcc391-69e5-4cc8-bbbc-47817715269e\" (UID: \"eedcc391-69e5-4cc8-bbbc-47817715269e\") " Oct 30 22:36:21.444047 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:36:21.443914 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eedcc391-69e5-4cc8-bbbc-47817715269e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eedcc391-69e5-4cc8-bbbc-47817715269e-maas-api-openshift-ca\") pod \"eedcc391-69e5-4cc8-bbbc-47817715269e\" (UID: \"eedcc391-69e5-4cc8-bbbc-47817715269e\") " Oct 30 22:36:21.444047 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:36:21.443948 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eedcc391-69e5-4cc8-bbbc-47817715269e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eedcc391-69e5-4cc8-bbbc-47817715269e-maas-api-xks-ca\") pod \"eedcc391-69e5-4cc8-bbbc-47817715269e\" (UID: \"eedcc391-69e5-4cc8-bbbc-47817715269e\") " Oct 30 22:36:21.444300 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:36:21.444278 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eedcc391-69e5-4cc8-bbbc-47817715269e-maas-api-openshift-ca" pod "eedcc391-69e5-4cc8-bbbc-47817715269e" (UID: "eedcc391-69e5-4cc8-bbbc-47817715269e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:36:21.445758 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:36:21.445737 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eedcc391-69e5-4cc8-bbbc-47817715269e-kube-api-access-v6svj" pod "eedcc391-69e5-4cc8-bbbc-47817715269e" (UID: "eedcc391-69e5-4cc8-bbbc-47817715269e"). InnerVolumeSpecName "kube-api-access-v6svj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:21.445863 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:36:21.445842 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eedcc391-69e5-4cc8-bbbc-47817715269e-maas-api-xks-ca" pod "eedcc391-69e5-4cc8-bbbc-47817715269e" (UID: "eedcc391-69e5-4cc8-bbbc-47817715269e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:36:21.546020 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:36:21.545933 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6svj\" (UniqueName: \"kubernetes.io/projected/eedcc391-69e5-4cc8-bbbc-47817715269e-kube-api-access-v6svj\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:21.546020 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:36:21.545972 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eedcc391-69e5-4cc8-bbbc-47817715269e-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:21.546020 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:36:21.545988 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eedcc391-69e5-4cc8-bbbc-47817715269e-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:22.213452 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:36:22.213418 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-563990-bzql6" Oct 30 22:36:22.213619 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:36:22.213415 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-563990-bzql6" event={"ID":"eedcc391-69e5-4cc8-bbbc-47817715269e","Type":"ContainerDied","Data":"db70df110df281c85fd5ecee16e54b5aa445ffb4f8d5a3d7e597a47775032d07"} Oct 30 22:36:22.213619 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:36:22.213546 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db70df110df281c85fd5ecee16e54b5aa445ffb4f8d5a3d7e597a47775032d07" Oct 30 22:36:30.144602 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:36:30.144564 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:36:43.620478 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:36:43.620438 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-563990-bzql6"] Oct 30 22:36:43.670963 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:36:43.670930 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-563990-bzql6"] Oct 30 22:36:44.415180 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:36:44.415142 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eedcc391-69e5-4cc8-bbbc-47817715269e" path="/var/lib/kubelet/pods/eedcc391-69e5-4cc8-bbbc-47817715269e/volumes" Oct 30 22:37:00.143919 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:37:00.143854 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:37:29.316069 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:29.316030 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-q2svb"] Oct 30 22:37:29.316608 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:29.316437 5748 cpu_manager.go:414] "containerMap: removing container" podUID="eedcc391-69e5-4cc8-bbbc-47817715269e" containerName="revoke-keys" Oct 30 22:37:29.316608 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:29.316449 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eedcc391-69e5-4cc8-bbbc-47817715269e" containerName="revoke-keys" Oct 30 22:37:29.316608 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:29.316543 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="eedcc391-69e5-4cc8-bbbc-47817715269e" containerName="revoke-keys" Oct 30 22:37:29.319883 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:29.319863 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-q2svb" Oct 30 22:37:29.374517 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:29.374489 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:37:29.424955 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:29.424927 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:37:29.425101 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:29.425036 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mdpxk\"" Oct 30 22:37:29.426363 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:29.426338 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-q2svb"] Oct 30 22:37:29.430398 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:29.430375 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44e06874-9787-4342-a499-58859e377a82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-q2svb\" (UID: \"44e06874-9787-4342-a499-58859e377a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-q2svb" Oct 30 22:37:29.430468 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:29.430413 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44e06874-9787-4342-a499-58859e377a82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-q2svb\" (UID: \"44e06874-9787-4342-a499-58859e377a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-q2svb" Oct 30 22:37:29.430554 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:29.430542 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-887rh\" (UniqueName: \"kubernetes.io/projected/44e06874-9787-4342-a499-58859e377a82-kube-api-access-887rh\") pod \"maas-api-revoke-sub-f03629db-q2svb\" (UID: \"44e06874-9787-4342-a499-58859e377a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-q2svb" Oct 30 22:37:29.476670 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:29.476644 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:37:29.531000 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:29.530974 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-887rh\" (UniqueName: \"kubernetes.io/projected/44e06874-9787-4342-a499-58859e377a82-kube-api-access-887rh\") pod \"maas-api-revoke-sub-f03629db-q2svb\" (UID: \"44e06874-9787-4342-a499-58859e377a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-q2svb" Oct 30 22:37:29.531118 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:29.531041 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44e06874-9787-4342-a499-58859e377a82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-q2svb\" (UID: \"44e06874-9787-4342-a499-58859e377a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-q2svb" Oct 30 22:37:29.531118 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:29.531066 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44e06874-9787-4342-a499-58859e377a82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-q2svb\" (UID: \"44e06874-9787-4342-a499-58859e377a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-q2svb" Oct 30 22:37:29.531665 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:29.531643 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44e06874-9787-4342-a499-58859e377a82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-q2svb\" (UID: \"44e06874-9787-4342-a499-58859e377a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-q2svb" Oct 30 22:37:29.533456 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:29.533433 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44e06874-9787-4342-a499-58859e377a82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-q2svb\" (UID: \"44e06874-9787-4342-a499-58859e377a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-q2svb" Oct 30 22:37:29.592892 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:29.592829 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-887rh\" (UniqueName: \"kubernetes.io/projected/44e06874-9787-4342-a499-58859e377a82-kube-api-access-887rh\") pod \"maas-api-revoke-sub-f03629db-q2svb\" (UID: \"44e06874-9787-4342-a499-58859e377a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-q2svb" Oct 30 22:37:29.631919 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:29.631871 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-q2svb" Oct 30 22:37:30.008600 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:30.008566 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-q2svb"] Oct 30 22:37:30.011556 ip-10-0-143-61 kubenswrapper[5748]: W1030 22:37:30.011529 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44e06874_9787_4342_a499_58859e377a82.slice/crio-afd8a53564bd38d239a1db9a24f6b7ed77f280edf7c3c805027478d431f357a1 WatchSource:0}: Error finding container afd8a53564bd38d239a1db9a24f6b7ed77f280edf7c3c805027478d431f357a1: Status 404 returned error can't find the container with id afd8a53564bd38d239a1db9a24f6b7ed77f280edf7c3c805027478d431f357a1 Oct 30 22:37:30.143268 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:37:30.143230 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:37:30.520065 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:30.520026 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-q2svb" event={"ID":"44e06874-9787-4342-a499-58859e377a82","Type":"ContainerStarted","Data":"5b795d09a0e327064ab0815e1f55f79ada8d6b340595449a138d1f78b53e69f4"} Oct 30 22:37:30.520454 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:30.520089 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-q2svb" event={"ID":"44e06874-9787-4342-a499-58859e377a82","Type":"ContainerStarted","Data":"afd8a53564bd38d239a1db9a24f6b7ed77f280edf7c3c805027478d431f357a1"} Oct 30 22:37:30.638913 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:30.638834 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-q2svb" podStartSLOduration=1.638820526 podStartE2EDuration="1.638820526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:37:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:37:30.637645744 +0000 UTC m=+4434.380166680" watchObservedRunningTime="2026-10-30 22:37:30.638820526 +0000 UTC m=+4434.381341461" Oct 30 22:37:31.526040 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:31.526002 5748 generic.go:356] "Generic (PLEG): container finished" podID="44e06874-9787-4342-a499-58859e377a82" containerID="5b795d09a0e327064ab0815e1f55f79ada8d6b340595449a138d1f78b53e69f4" exitCode=0 Oct 30 22:37:31.526460 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:31.526087 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-q2svb" event={"ID":"44e06874-9787-4342-a499-58859e377a82","Type":"ContainerDied","Data":"5b795d09a0e327064ab0815e1f55f79ada8d6b340595449a138d1f78b53e69f4"} Oct 30 22:37:32.662138 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:32.662114 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-q2svb" Oct 30 22:37:32.759915 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:32.759870 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44e06874-9787-4342-a499-58859e377a82-kube-api-access-887rh\" (UniqueName: \"kubernetes.io/projected/44e06874-9787-4342-a499-58859e377a82-kube-api-access-887rh\") pod \"44e06874-9787-4342-a499-58859e377a82\" (UID: \"44e06874-9787-4342-a499-58859e377a82\") " Oct 30 22:37:32.760058 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:32.759989 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44e06874-9787-4342-a499-58859e377a82-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44e06874-9787-4342-a499-58859e377a82-maas-api-openshift-ca\") pod \"44e06874-9787-4342-a499-58859e377a82\" (UID: \"44e06874-9787-4342-a499-58859e377a82\") " Oct 30 22:37:32.760113 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:32.760053 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44e06874-9787-4342-a499-58859e377a82-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44e06874-9787-4342-a499-58859e377a82-maas-api-xks-ca\") pod \"44e06874-9787-4342-a499-58859e377a82\" (UID: \"44e06874-9787-4342-a499-58859e377a82\") " Oct 30 22:37:32.760334 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:32.760311 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44e06874-9787-4342-a499-58859e377a82-maas-api-openshift-ca" pod "44e06874-9787-4342-a499-58859e377a82" (UID: "44e06874-9787-4342-a499-58859e377a82"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:37:32.762004 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:32.761980 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44e06874-9787-4342-a499-58859e377a82-maas-api-xks-ca" pod "44e06874-9787-4342-a499-58859e377a82" (UID: "44e06874-9787-4342-a499-58859e377a82"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:37:32.762133 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:32.762066 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44e06874-9787-4342-a499-58859e377a82-kube-api-access-887rh" pod "44e06874-9787-4342-a499-58859e377a82" (UID: "44e06874-9787-4342-a499-58859e377a82"). InnerVolumeSpecName "kube-api-access-887rh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:37:32.861441 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:32.861378 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44e06874-9787-4342-a499-58859e377a82-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:37:32.861441 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:32.861406 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44e06874-9787-4342-a499-58859e377a82-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:37:32.861441 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:32.861416 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-887rh\" (UniqueName: \"kubernetes.io/projected/44e06874-9787-4342-a499-58859e377a82-kube-api-access-887rh\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:37:33.536534 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:33.536505 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-q2svb" Oct 30 22:37:33.536690 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:33.536507 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-q2svb" event={"ID":"44e06874-9787-4342-a499-58859e377a82","Type":"ContainerDied","Data":"afd8a53564bd38d239a1db9a24f6b7ed77f280edf7c3c805027478d431f357a1"} Oct 30 22:37:33.536690 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:33.536631 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afd8a53564bd38d239a1db9a24f6b7ed77f280edf7c3c805027478d431f357a1" Oct 30 22:37:48.720922 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:48.719945 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ltwfq"] Oct 30 22:37:48.720922 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:48.720556 5748 cpu_manager.go:414] "containerMap: removing container" podUID="44e06874-9787-4342-a499-58859e377a82" containerName="revoke-keys" Oct 30 22:37:48.720922 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:48.720576 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44e06874-9787-4342-a499-58859e377a82" containerName="revoke-keys" Oct 30 22:37:48.720922 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:48.720711 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="44e06874-9787-4342-a499-58859e377a82" containerName="revoke-keys" Oct 30 22:37:48.741919 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:48.741135 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ltwfq" Oct 30 22:37:48.785648 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:48.785597 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ltwfq"] Oct 30 22:37:48.820680 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:48.820366 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aedacd37-99b3-4a47-adae-5aebf07d4e0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-ltwfq\" (UID: \"aedacd37-99b3-4a47-adae-5aebf07d4e0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ltwfq" Oct 30 22:37:48.820680 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:48.820475 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aedacd37-99b3-4a47-adae-5aebf07d4e0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-ltwfq\" (UID: \"aedacd37-99b3-4a47-adae-5aebf07d4e0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ltwfq" Oct 30 22:37:48.820680 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:48.820517 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkvvp\" (UniqueName: \"kubernetes.io/projected/aedacd37-99b3-4a47-adae-5aebf07d4e0d-kube-api-access-xkvvp\") pod \"maas-api-revoke-sub-0165d7d2-ltwfq\" (UID: \"aedacd37-99b3-4a47-adae-5aebf07d4e0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ltwfq" Oct 30 22:37:48.836922 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:48.835151 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:37:48.836922 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:48.835279 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mdpxk\"" Oct 30 22:37:48.836922 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:48.835342 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:37:48.875918 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:48.856048 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:37:48.924781 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:48.924750 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aedacd37-99b3-4a47-adae-5aebf07d4e0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-ltwfq\" (UID: \"aedacd37-99b3-4a47-adae-5aebf07d4e0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ltwfq" Oct 30 22:37:48.937966 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:48.925063 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkvvp\" (UniqueName: \"kubernetes.io/projected/aedacd37-99b3-4a47-adae-5aebf07d4e0d-kube-api-access-xkvvp\") pod \"maas-api-revoke-sub-0165d7d2-ltwfq\" (UID: \"aedacd37-99b3-4a47-adae-5aebf07d4e0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ltwfq" Oct 30 22:37:48.938344 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:48.938321 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aedacd37-99b3-4a47-adae-5aebf07d4e0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-ltwfq\" (UID: \"aedacd37-99b3-4a47-adae-5aebf07d4e0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ltwfq" Oct 30 22:37:48.938939 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:48.925670 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aedacd37-99b3-4a47-adae-5aebf07d4e0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-ltwfq\" (UID: \"aedacd37-99b3-4a47-adae-5aebf07d4e0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ltwfq" Oct 30 22:37:48.961438 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:48.961408 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aedacd37-99b3-4a47-adae-5aebf07d4e0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-ltwfq\" (UID: \"aedacd37-99b3-4a47-adae-5aebf07d4e0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ltwfq" Oct 30 22:37:48.996616 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:48.996554 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkvvp\" (UniqueName: \"kubernetes.io/projected/aedacd37-99b3-4a47-adae-5aebf07d4e0d-kube-api-access-xkvvp\") pod \"maas-api-revoke-sub-0165d7d2-ltwfq\" (UID: \"aedacd37-99b3-4a47-adae-5aebf07d4e0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ltwfq" Oct 30 22:37:49.087182 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:49.087149 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ltwfq" Oct 30 22:37:49.569922 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:49.552394 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ltwfq"] Oct 30 22:37:49.651031 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:49.651002 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ltwfq" event={"ID":"aedacd37-99b3-4a47-adae-5aebf07d4e0d","Type":"ContainerStarted","Data":"3253fb16364d115c1ef890d452a32e252587a8719a94d5b2967ce7715d2639a4"} Oct 30 22:37:50.656687 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:50.656656 5748 generic.go:356] "Generic (PLEG): container finished" podID="aedacd37-99b3-4a47-adae-5aebf07d4e0d" containerID="4ca261494180beb264352f6b2dc10d795f5838952bd90e153235f978aca77b37" exitCode=0 Oct 30 22:37:50.657047 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:50.656736 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ltwfq" event={"ID":"aedacd37-99b3-4a47-adae-5aebf07d4e0d","Type":"ContainerDied","Data":"4ca261494180beb264352f6b2dc10d795f5838952bd90e153235f978aca77b37"} Oct 30 22:37:51.794043 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:51.794018 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ltwfq" Oct 30 22:37:51.797236 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:51.797213 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aedacd37-99b3-4a47-adae-5aebf07d4e0d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aedacd37-99b3-4a47-adae-5aebf07d4e0d-maas-api-openshift-ca\") pod \"aedacd37-99b3-4a47-adae-5aebf07d4e0d\" (UID: \"aedacd37-99b3-4a47-adae-5aebf07d4e0d\") " Oct 30 22:37:51.797361 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:51.797259 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aedacd37-99b3-4a47-adae-5aebf07d4e0d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aedacd37-99b3-4a47-adae-5aebf07d4e0d-maas-api-xks-ca\") pod \"aedacd37-99b3-4a47-adae-5aebf07d4e0d\" (UID: \"aedacd37-99b3-4a47-adae-5aebf07d4e0d\") " Oct 30 22:37:51.797361 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:51.797282 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aedacd37-99b3-4a47-adae-5aebf07d4e0d-kube-api-access-xkvvp\" (UniqueName: \"kubernetes.io/projected/aedacd37-99b3-4a47-adae-5aebf07d4e0d-kube-api-access-xkvvp\") pod \"aedacd37-99b3-4a47-adae-5aebf07d4e0d\" (UID: \"aedacd37-99b3-4a47-adae-5aebf07d4e0d\") " Oct 30 22:37:51.797584 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:51.797559 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aedacd37-99b3-4a47-adae-5aebf07d4e0d-maas-api-openshift-ca" pod "aedacd37-99b3-4a47-adae-5aebf07d4e0d" (UID: "aedacd37-99b3-4a47-adae-5aebf07d4e0d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:37:51.799231 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:51.799206 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aedacd37-99b3-4a47-adae-5aebf07d4e0d-maas-api-xks-ca" pod "aedacd37-99b3-4a47-adae-5aebf07d4e0d" (UID: "aedacd37-99b3-4a47-adae-5aebf07d4e0d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:37:51.799325 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:51.799284 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aedacd37-99b3-4a47-adae-5aebf07d4e0d-kube-api-access-xkvvp" pod "aedacd37-99b3-4a47-adae-5aebf07d4e0d" (UID: "aedacd37-99b3-4a47-adae-5aebf07d4e0d"). InnerVolumeSpecName "kube-api-access-xkvvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:37:51.898493 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:51.898469 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aedacd37-99b3-4a47-adae-5aebf07d4e0d-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:37:51.898493 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:51.898492 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aedacd37-99b3-4a47-adae-5aebf07d4e0d-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:37:51.898639 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:51.898503 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkvvp\" (UniqueName: \"kubernetes.io/projected/aedacd37-99b3-4a47-adae-5aebf07d4e0d-kube-api-access-xkvvp\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:37:52.666485 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:52.666456 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ltwfq" Oct 30 22:37:52.666667 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:52.666488 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ltwfq" event={"ID":"aedacd37-99b3-4a47-adae-5aebf07d4e0d","Type":"ContainerDied","Data":"3253fb16364d115c1ef890d452a32e252587a8719a94d5b2967ce7715d2639a4"} Oct 30 22:37:52.666667 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:37:52.666534 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3253fb16364d115c1ef890d452a32e252587a8719a94d5b2967ce7715d2639a4" Oct 30 22:38:00.143728 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:38:00.143685 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:38:00.570716 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:38:00.570634 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fdzj6"] Oct 30 22:38:00.571041 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:38:00.571027 5748 cpu_manager.go:414] "containerMap: removing container" podUID="aedacd37-99b3-4a47-adae-5aebf07d4e0d" containerName="revoke-keys" Oct 30 22:38:00.571096 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:38:00.571041 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aedacd37-99b3-4a47-adae-5aebf07d4e0d" containerName="revoke-keys" Oct 30 22:38:00.571138 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:38:00.571131 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="aedacd37-99b3-4a47-adae-5aebf07d4e0d" containerName="revoke-keys" Oct 30 22:38:00.574244 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:38:00.574227 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fdzj6" Oct 30 22:38:00.627369 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:38:00.627347 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:38:00.677415 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:38:00.677387 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:38:00.677576 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:38:00.677465 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:38:00.677685 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:38:00.677669 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mdpxk\"" Oct 30 22:38:00.677748 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:38:00.677715 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98eb4d95-60dc-419e-a650-14b87682b524-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-fdzj6\" (UID: \"98eb4d95-60dc-419e-a650-14b87682b524\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fdzj6" Oct 30 22:38:00.677844 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:38:00.677826 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98eb4d95-60dc-419e-a650-14b87682b524-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-fdzj6\" (UID: \"98eb4d95-60dc-419e-a650-14b87682b524\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fdzj6" Oct 30 22:38:00.677932 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:38:00.677871 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg86s\" (UniqueName: \"kubernetes.io/projected/98eb4d95-60dc-419e-a650-14b87682b524-kube-api-access-lg86s\") pod \"maas-api-revoke-sub-6b19338f-fdzj6\" (UID: \"98eb4d95-60dc-419e-a650-14b87682b524\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fdzj6" Oct 30 22:38:00.679091 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:38:00.679071 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fdzj6"] Oct 30 22:38:00.778572 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:38:00.778543 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98eb4d95-60dc-419e-a650-14b87682b524-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-fdzj6\" (UID: \"98eb4d95-60dc-419e-a650-14b87682b524\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fdzj6" Oct 30 22:38:00.778726 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:38:00.778614 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98eb4d95-60dc-419e-a650-14b87682b524-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-fdzj6\" (UID: \"98eb4d95-60dc-419e-a650-14b87682b524\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fdzj6" Oct 30 22:38:00.778726 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:38:00.778644 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg86s\" (UniqueName: \"kubernetes.io/projected/98eb4d95-60dc-419e-a650-14b87682b524-kube-api-access-lg86s\") pod \"maas-api-revoke-sub-6b19338f-fdzj6\" (UID: \"98eb4d95-60dc-419e-a650-14b87682b524\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fdzj6" Oct 30 22:38:00.779242 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:38:00.779222 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98eb4d95-60dc-419e-a650-14b87682b524-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-fdzj6\" (UID: \"98eb4d95-60dc-419e-a650-14b87682b524\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fdzj6" Oct 30 22:38:00.780945 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:38:00.780925 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98eb4d95-60dc-419e-a650-14b87682b524-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-fdzj6\" (UID: \"98eb4d95-60dc-419e-a650-14b87682b524\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fdzj6" Oct 30 22:38:00.840162 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:38:00.840090 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg86s\" (UniqueName: \"kubernetes.io/projected/98eb4d95-60dc-419e-a650-14b87682b524-kube-api-access-lg86s\") pod \"maas-api-revoke-sub-6b19338f-fdzj6\" (UID: \"98eb4d95-60dc-419e-a650-14b87682b524\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fdzj6" Oct 30 22:38:00.887423 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:38:00.887383 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fdzj6" Oct 30 22:38:01.258249 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:38:01.258223 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fdzj6"] Oct 30 22:38:01.261100 ip-10-0-143-61 kubenswrapper[5748]: W1030 22:38:01.261071 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98eb4d95_60dc_419e_a650_14b87682b524.slice/crio-3a070b6d8d022c4dfbb30e9fc7fe4f5d483ffd785db9ce9204ec4ff506f43187 WatchSource:0}: Error finding container 3a070b6d8d022c4dfbb30e9fc7fe4f5d483ffd785db9ce9204ec4ff506f43187: Status 404 returned error can't find the container with id 3a070b6d8d022c4dfbb30e9fc7fe4f5d483ffd785db9ce9204ec4ff506f43187 Oct 30 22:38:01.711591 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:38:01.711555 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fdzj6" event={"ID":"98eb4d95-60dc-419e-a650-14b87682b524","Type":"ContainerStarted","Data":"beb3da2467bd62ce8495ab69b2ddbf99b5d412be2d4008d8e7f6d871b8bdc1fa"} Oct 30 22:38:01.711753 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:38:01.711614 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fdzj6" event={"ID":"98eb4d95-60dc-419e-a650-14b87682b524","Type":"ContainerStarted","Data":"3a070b6d8d022c4dfbb30e9fc7fe4f5d483ffd785db9ce9204ec4ff506f43187"} Oct 30 22:38:01.830562 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:38:01.830469 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fdzj6" podStartSLOduration=1.8304514219999999 podStartE2EDuration="1.830451422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:38:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:38:01.829704465 +0000 UTC m=+4465.572225400" watchObservedRunningTime="2026-10-30 22:38:01.830451422 +0000 UTC m=+4465.572972351" Oct 30 22:38:02.717152 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:38:02.717109 5748 generic.go:356] "Generic (PLEG): container finished" podID="98eb4d95-60dc-419e-a650-14b87682b524" containerID="beb3da2467bd62ce8495ab69b2ddbf99b5d412be2d4008d8e7f6d871b8bdc1fa" exitCode=0 Oct 30 22:38:02.717591 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:38:02.717154 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fdzj6" event={"ID":"98eb4d95-60dc-419e-a650-14b87682b524","Type":"ContainerDied","Data":"beb3da2467bd62ce8495ab69b2ddbf99b5d412be2d4008d8e7f6d871b8bdc1fa"} Oct 30 22:38:03.858712 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:38:03.858685 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fdzj6" Oct 30 22:38:03.906835 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:38:03.906798 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98eb4d95-60dc-419e-a650-14b87682b524-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98eb4d95-60dc-419e-a650-14b87682b524-maas-api-openshift-ca\") pod \"98eb4d95-60dc-419e-a650-14b87682b524\" (UID: \"98eb4d95-60dc-419e-a650-14b87682b524\") " Oct 30 22:38:03.907033 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:38:03.906858 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98eb4d95-60dc-419e-a650-14b87682b524-kube-api-access-lg86s\" (UniqueName: \"kubernetes.io/projected/98eb4d95-60dc-419e-a650-14b87682b524-kube-api-access-lg86s\") pod \"98eb4d95-60dc-419e-a650-14b87682b524\" (UID: \"98eb4d95-60dc-419e-a650-14b87682b524\") " Oct 30 22:38:03.907033 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:38:03.906881 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98eb4d95-60dc-419e-a650-14b87682b524-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98eb4d95-60dc-419e-a650-14b87682b524-maas-api-xks-ca\") pod \"98eb4d95-60dc-419e-a650-14b87682b524\" (UID: \"98eb4d95-60dc-419e-a650-14b87682b524\") " Oct 30 22:38:03.907301 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:38:03.907274 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98eb4d95-60dc-419e-a650-14b87682b524-maas-api-openshift-ca" pod "98eb4d95-60dc-419e-a650-14b87682b524" (UID: "98eb4d95-60dc-419e-a650-14b87682b524"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:38:03.909127 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:38:03.909100 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98eb4d95-60dc-419e-a650-14b87682b524-kube-api-access-lg86s" pod "98eb4d95-60dc-419e-a650-14b87682b524" (UID: "98eb4d95-60dc-419e-a650-14b87682b524"). InnerVolumeSpecName "kube-api-access-lg86s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:38:03.909127 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:38:03.909104 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98eb4d95-60dc-419e-a650-14b87682b524-maas-api-xks-ca" pod "98eb4d95-60dc-419e-a650-14b87682b524" (UID: "98eb4d95-60dc-419e-a650-14b87682b524"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:38:04.008198 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:38:04.008112 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98eb4d95-60dc-419e-a650-14b87682b524-maas-api-openshift-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:38:04.008198 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:38:04.008140 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg86s\" (UniqueName: \"kubernetes.io/projected/98eb4d95-60dc-419e-a650-14b87682b524-kube-api-access-lg86s\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:38:04.008198 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:38:04.008150 5748 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98eb4d95-60dc-419e-a650-14b87682b524-maas-api-xks-ca\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:38:04.734879 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:38:04.734794 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fdzj6" Oct 30 22:38:04.735060 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:38:04.734798 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fdzj6" event={"ID":"98eb4d95-60dc-419e-a650-14b87682b524","Type":"ContainerDied","Data":"3a070b6d8d022c4dfbb30e9fc7fe4f5d483ffd785db9ce9204ec4ff506f43187"} Oct 30 22:38:04.735060 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:38:04.734954 5748 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a070b6d8d022c4dfbb30e9fc7fe4f5d483ffd785db9ce9204ec4ff506f43187" Oct 30 22:38:30.143981 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:38:30.143944 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:38:34.955154 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:38:34.955120 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-zq226_d611ba76-37b9-4cd6-aac5-14dfa4cf136b/manager/0.log" Oct 30 22:38:35.853313 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:38:35.853280 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-nklnt_62a7105e-4256-4b8d-8117-25d9a9c8a506/server/0.log" Oct 30 22:38:37.063440 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:38:37.063409 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j4v8r_3df036ff-1c00-4ce2-9e86-8acbeedf6c82/ovn-acl-logging/0.log" Oct 30 22:38:37.086979 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:38:37.086954 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j4v8r_3df036ff-1c00-4ce2-9e86-8acbeedf6c82/ovn-acl-logging/0.log" Oct 30 22:38:37.932146 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:38:37.932111 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-ltwfq_aedacd37-99b3-4a47-adae-5aebf07d4e0d/revoke-keys/0.log" Oct 30 22:38:38.167871 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:38:38.167834 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-fdzj6_98eb4d95-60dc-419e-a650-14b87682b524/revoke-keys/0.log" Oct 30 22:38:38.400022 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:38:38.399990 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-q2svb_44e06874-9787-4342-a499-58859e377a82/revoke-keys/0.log" Oct 30 22:38:51.449360 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:38:51.449284 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-fm72c_8b9b71d4-264b-4666-bfe3-ff0dcffafc85/kuadrant-console-plugin/0.log" Oct 30 22:38:53.923278 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:38:53.923250 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6bcd447999-f4qch_e692bffa-b34f-4ce9-9db8-6667ed013afb/kube-auth-proxy/0.log" Oct 30 22:38:54.891314 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:38:54.891281 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-948nx_f411f535-b1bc-408b-afb9-3299fc9d5bd6/istio-proxy/0.log" Oct 30 22:38:55.169546 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:38:55.169461 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-98fnd_d1c1a2b1-f312-47b1-b437-2d8f1760eea0/istio-proxy/0.log" Oct 30 22:38:56.161830 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:38:56.161798 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-t7djv_818838b2-f54f-4591-b855-73014d18a06e/istio-proxy/0.log" Oct 30 22:39:00.144485 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:39:00.144452 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:39:30.143664 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:39:30.143627 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:39:31.865258 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:39:31.865227 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-zq226_d611ba76-37b9-4cd6-aac5-14dfa4cf136b/manager/0.log" Oct 30 22:39:32.507238 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:39:32.507209 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-nklnt_62a7105e-4256-4b8d-8117-25d9a9c8a506/server/0.log" Oct 30 22:39:34.388706 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:39:34.388674 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-ltwfq_aedacd37-99b3-4a47-adae-5aebf07d4e0d/revoke-keys/0.log" Oct 30 22:39:34.637596 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:39:34.637566 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-fdzj6_98eb4d95-60dc-419e-a650-14b87682b524/revoke-keys/0.log" Oct 30 22:39:34.872506 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:39:34.872429 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-q2svb_44e06874-9787-4342-a499-58859e377a82/revoke-keys/0.log" Oct 30 22:39:37.260340 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:39:37.260309 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-fm72c_8b9b71d4-264b-4666-bfe3-ff0dcffafc85/kuadrant-console-plugin/0.log" Oct 30 22:39:39.485174 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:39:39.485134 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6bcd447999-f4qch_e692bffa-b34f-4ce9-9db8-6667ed013afb/kube-auth-proxy/0.log" Oct 30 22:39:50.524422 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:39:50.524389 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-948nx_f411f535-b1bc-408b-afb9-3299fc9d5bd6/istio-proxy/0.log" Oct 30 22:39:50.864472 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:39:50.864372 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-98fnd_d1c1a2b1-f312-47b1-b437-2d8f1760eea0/istio-proxy/0.log" Oct 30 22:39:51.869303 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:39:51.869271 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-t7djv_818838b2-f54f-4591-b855-73014d18a06e/istio-proxy/0.log" Oct 30 22:40:00.144240 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:40:00.144194 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:40:30.143012 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:40:30.142975 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:40:39.034494 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:40:39.034457 5748 scope.go:122] "RemoveContainer" containerID="8bc066a35311bff869c7fc1722ed7db5e9513c241e70891d696f50d95dfff2ca" Oct 30 22:41:00.143141 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:41:00.143102 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:41:30.143938 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:41:30.143884 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:42:00.143717 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:42:00.143639 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:42:30.144053 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:42:30.144015 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:42:33.048263 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:42:33.048226 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-q2svb"] Oct 30 22:42:33.098406 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:42:33.098374 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-q2svb"] Oct 30 22:42:34.415761 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:42:34.415729 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44e06874-9787-4342-a499-58859e377a82" path="/var/lib/kubelet/pods/44e06874-9787-4342-a499-58859e377a82/volumes" Oct 30 22:42:39.059883 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:42:39.059851 5748 scope.go:122] "RemoveContainer" containerID="9fb67fc393b6118cef2908b8c98505510ec85f3051add790de229c2b2d1fdd27" Oct 30 22:42:52.054946 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:42:52.054913 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ltwfq"] Oct 30 22:42:52.105234 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:42:52.105197 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ltwfq"] Oct 30 22:42:52.415870 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:42:52.415838 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aedacd37-99b3-4a47-adae-5aebf07d4e0d" path="/var/lib/kubelet/pods/aedacd37-99b3-4a47-adae-5aebf07d4e0d/volumes" Oct 30 22:42:59.575394 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:42:59.575356 5748 ???:1] "http: TLS handshake error from 10.0.143.61:60676: EOF" Oct 30 22:42:59.583619 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:42:59.583594 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6nncz_6c9c626a-52a7-466e-b744-f4bd79112fd9/global-pull-secret-syncer/0.log" Oct 30 22:43:00.144348 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:43:00.144311 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:43:03.222989 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:03.222955 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-75hkd_454a9788-f96e-453d-a83c-63c550c7d96d/konnectivity-agent/0.log" Oct 30 22:43:04.047694 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:04.047658 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fdzj6"] Oct 30 22:43:04.098029 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:04.097992 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fdzj6"] Oct 30 22:43:04.415282 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:04.415250 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98eb4d95-60dc-419e-a650-14b87682b524" path="/var/lib/kubelet/pods/98eb4d95-60dc-419e-a650-14b87682b524/volumes" Oct 30 22:43:04.973269 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:04.973218 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-61.us-west-1.compute.internal_f05411f241c17e725b228582bf16173f/haproxy/0.log" Oct 30 22:43:09.219023 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:09.218993 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-fm72c_8b9b71d4-264b-4666-bfe3-ff0dcffafc85/kuadrant-console-plugin/0.log" Oct 30 22:43:16.615791 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:16.615759 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-kqxdq_166e5f35-0987-4eef-917a-a4772acba82f/kube-state-metrics/0.log" Oct 30 22:43:17.107477 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:17.107437 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-kqxdq_166e5f35-0987-4eef-917a-a4772acba82f/kube-rbac-proxy-main/0.log" Oct 30 22:43:17.529588 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:17.529555 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-kqxdq_166e5f35-0987-4eef-917a-a4772acba82f/kube-rbac-proxy-self/0.log" Oct 30 22:43:17.923506 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:17.923475 5748 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mh4rx/perf-node-gather-daemonset-qzh8d"] Oct 30 22:43:17.924098 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:17.924070 5748 cpu_manager.go:414] "containerMap: removing container" podUID="98eb4d95-60dc-419e-a650-14b87682b524" containerName="revoke-keys" Oct 30 22:43:17.924098 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:17.924091 5748 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98eb4d95-60dc-419e-a650-14b87682b524" containerName="revoke-keys" Oct 30 22:43:17.924273 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:17.924230 5748 memory_manager.go:365] "RemoveStaleState removing state" podUID="98eb4d95-60dc-419e-a650-14b87682b524" containerName="revoke-keys" Oct 30 22:43:17.927182 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:17.927162 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mh4rx/perf-node-gather-daemonset-qzh8d" Oct 30 22:43:17.954239 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:17.954205 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5dcb975b97-crpzf_4372da07-b518-42df-bbd7-f148396bba6b/metrics-server/0.log" Oct 30 22:43:17.999285 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:17.999262 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3d6315bf-6f49-4ea2-8b68-feeaf73961fe-sys\") pod \"perf-node-gather-daemonset-qzh8d\" (UID: \"3d6315bf-6f49-4ea2-8b68-feeaf73961fe\") " pod="openshift-must-gather-mh4rx/perf-node-gather-daemonset-qzh8d" Oct 30 22:43:17.999411 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:17.999294 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4278c\" (UniqueName: \"kubernetes.io/projected/3d6315bf-6f49-4ea2-8b68-feeaf73961fe-kube-api-access-4278c\") pod \"perf-node-gather-daemonset-qzh8d\" (UID: \"3d6315bf-6f49-4ea2-8b68-feeaf73961fe\") " pod="openshift-must-gather-mh4rx/perf-node-gather-daemonset-qzh8d" Oct 30 22:43:17.999411 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:17.999314 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3d6315bf-6f49-4ea2-8b68-feeaf73961fe-podres\") pod \"perf-node-gather-daemonset-qzh8d\" (UID: \"3d6315bf-6f49-4ea2-8b68-feeaf73961fe\") " pod="openshift-must-gather-mh4rx/perf-node-gather-daemonset-qzh8d" Oct 30 22:43:17.999411 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:17.999334 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3d6315bf-6f49-4ea2-8b68-feeaf73961fe-proc\") pod \"perf-node-gather-daemonset-qzh8d\" (UID: \"3d6315bf-6f49-4ea2-8b68-feeaf73961fe\") " pod="openshift-must-gather-mh4rx/perf-node-gather-daemonset-qzh8d" Oct 30 22:43:17.999526 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:17.999500 5748 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3d6315bf-6f49-4ea2-8b68-feeaf73961fe-lib-modules\") pod \"perf-node-gather-daemonset-qzh8d\" (UID: \"3d6315bf-6f49-4ea2-8b68-feeaf73961fe\") " pod="openshift-must-gather-mh4rx/perf-node-gather-daemonset-qzh8d" Oct 30 22:43:18.002769 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:18.002753 5748 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mh4rx\"/\"default-dockercfg-2gdt5\"" Oct 30 22:43:18.002834 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:18.002795 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mh4rx\"/\"kube-root-ca.crt\"" Oct 30 22:43:18.002882 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:18.002839 5748 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mh4rx\"/\"openshift-service-ca.crt\"" Oct 30 22:43:18.036723 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:18.036694 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mh4rx/perf-node-gather-daemonset-qzh8d"] Oct 30 22:43:18.100449 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:18.100418 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4278c\" (UniqueName: \"kubernetes.io/projected/3d6315bf-6f49-4ea2-8b68-feeaf73961fe-kube-api-access-4278c\") pod \"perf-node-gather-daemonset-qzh8d\" (UID: \"3d6315bf-6f49-4ea2-8b68-feeaf73961fe\") " pod="openshift-must-gather-mh4rx/perf-node-gather-daemonset-qzh8d" Oct 30 22:43:18.100449 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:18.100452 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3d6315bf-6f49-4ea2-8b68-feeaf73961fe-podres\") pod \"perf-node-gather-daemonset-qzh8d\" (UID: \"3d6315bf-6f49-4ea2-8b68-feeaf73961fe\") " pod="openshift-must-gather-mh4rx/perf-node-gather-daemonset-qzh8d" Oct 30 22:43:18.100688 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:18.100471 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3d6315bf-6f49-4ea2-8b68-feeaf73961fe-proc\") pod \"perf-node-gather-daemonset-qzh8d\" (UID: \"3d6315bf-6f49-4ea2-8b68-feeaf73961fe\") " pod="openshift-must-gather-mh4rx/perf-node-gather-daemonset-qzh8d" Oct 30 22:43:18.100688 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:18.100539 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3d6315bf-6f49-4ea2-8b68-feeaf73961fe-lib-modules\") pod \"perf-node-gather-daemonset-qzh8d\" (UID: \"3d6315bf-6f49-4ea2-8b68-feeaf73961fe\") " pod="openshift-must-gather-mh4rx/perf-node-gather-daemonset-qzh8d" Oct 30 22:43:18.100688 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:18.100565 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3d6315bf-6f49-4ea2-8b68-feeaf73961fe-podres\") pod \"perf-node-gather-daemonset-qzh8d\" (UID: \"3d6315bf-6f49-4ea2-8b68-feeaf73961fe\") " pod="openshift-must-gather-mh4rx/perf-node-gather-daemonset-qzh8d" Oct 30 22:43:18.100688 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:18.100570 5748 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3d6315bf-6f49-4ea2-8b68-feeaf73961fe-sys\") pod \"perf-node-gather-daemonset-qzh8d\" (UID: \"3d6315bf-6f49-4ea2-8b68-feeaf73961fe\") " pod="openshift-must-gather-mh4rx/perf-node-gather-daemonset-qzh8d" Oct 30 22:43:18.100688 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:18.100632 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3d6315bf-6f49-4ea2-8b68-feeaf73961fe-proc\") pod \"perf-node-gather-daemonset-qzh8d\" (UID: \"3d6315bf-6f49-4ea2-8b68-feeaf73961fe\") " pod="openshift-must-gather-mh4rx/perf-node-gather-daemonset-qzh8d" Oct 30 22:43:18.100688 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:18.100655 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3d6315bf-6f49-4ea2-8b68-feeaf73961fe-sys\") pod \"perf-node-gather-daemonset-qzh8d\" (UID: \"3d6315bf-6f49-4ea2-8b68-feeaf73961fe\") " pod="openshift-must-gather-mh4rx/perf-node-gather-daemonset-qzh8d" Oct 30 22:43:18.100961 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:18.100722 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3d6315bf-6f49-4ea2-8b68-feeaf73961fe-lib-modules\") pod \"perf-node-gather-daemonset-qzh8d\" (UID: \"3d6315bf-6f49-4ea2-8b68-feeaf73961fe\") " pod="openshift-must-gather-mh4rx/perf-node-gather-daemonset-qzh8d" Oct 30 22:43:18.162779 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:18.162756 5748 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4278c\" (UniqueName: \"kubernetes.io/projected/3d6315bf-6f49-4ea2-8b68-feeaf73961fe-kube-api-access-4278c\") pod \"perf-node-gather-daemonset-qzh8d\" (UID: \"3d6315bf-6f49-4ea2-8b68-feeaf73961fe\") " pod="openshift-must-gather-mh4rx/perf-node-gather-daemonset-qzh8d" Oct 30 22:43:18.238855 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:18.238766 5748 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mh4rx/perf-node-gather-daemonset-qzh8d" Oct 30 22:43:18.385004 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:18.384961 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-2r74x_f0d8d720-b6f1-4600-970f-8191318c3e3e/monitoring-plugin/0.log" Oct 30 22:43:18.614379 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:18.614145 5748 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mh4rx/perf-node-gather-daemonset-qzh8d"] Oct 30 22:43:18.617213 ip-10-0-143-61 kubenswrapper[5748]: W1030 22:43:18.617186 5748 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3d6315bf_6f49_4ea2_8b68_feeaf73961fe.slice/crio-9502a058c0e636ca0bfa1494ed8769062944fd9136c3b42565b56d435ac15b99 WatchSource:0}: Error finding container 9502a058c0e636ca0bfa1494ed8769062944fd9136c3b42565b56d435ac15b99: Status 404 returned error can't find the container with id 9502a058c0e636ca0bfa1494ed8769062944fd9136c3b42565b56d435ac15b99 Oct 30 22:43:19.178694 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:19.178658 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mh4rx/perf-node-gather-daemonset-qzh8d" event={"ID":"3d6315bf-6f49-4ea2-8b68-feeaf73961fe","Type":"ContainerStarted","Data":"4704963b8dcb088b6592439863ab5325187b606f36786c3ef4ed84126d0ebf30"} Oct 30 22:43:19.179148 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:19.178716 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mh4rx/perf-node-gather-daemonset-qzh8d" event={"ID":"3d6315bf-6f49-4ea2-8b68-feeaf73961fe","Type":"ContainerStarted","Data":"9502a058c0e636ca0bfa1494ed8769062944fd9136c3b42565b56d435ac15b99"} Oct 30 22:43:19.179148 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:19.178847 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mh4rx/perf-node-gather-daemonset-qzh8d" Oct 30 22:43:19.297215 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:19.297161 5748 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-mh4rx/perf-node-gather-daemonset-qzh8d" podStartSLOduration=2.297146686 podStartE2EDuration="2.297146686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:43:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:43:19.296268805 +0000 UTC m=+4783.038789742" watchObservedRunningTime="2026-10-30 22:43:19.297146686 +0000 UTC m=+4783.039667661" Oct 30 22:43:21.367759 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:21.367727 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rlt89_1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979/node-exporter/0.log" Oct 30 22:43:21.783700 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:21.783665 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rlt89_1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979/kube-rbac-proxy/0.log" Oct 30 22:43:22.200831 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:22.200804 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rlt89_1a1e5944-c376-4bf1-b8ea-ae4d1b6a2979/init-textfile/0.log" Oct 30 22:43:22.625234 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:22.625148 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-jfpf9_ca9f1917-07a8-4ea6-8b59-4f2f99429dfa/kube-rbac-proxy-main/0.log" Oct 30 22:43:23.082152 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:23.082117 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-jfpf9_ca9f1917-07a8-4ea6-8b59-4f2f99429dfa/kube-rbac-proxy-self/0.log" Oct 30 22:43:23.527086 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:23.527057 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-jfpf9_ca9f1917-07a8-4ea6-8b59-4f2f99429dfa/openshift-state-metrics/0.log" Oct 30 22:43:23.961940 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:23.961876 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b287c83b-aa37-4063-9782-b62b202832d5/prometheus/0.log" Oct 30 22:43:24.380474 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:24.380386 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b287c83b-aa37-4063-9782-b62b202832d5/config-reloader/0.log" Oct 30 22:43:24.804497 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:24.804465 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b287c83b-aa37-4063-9782-b62b202832d5/thanos-sidecar/0.log" Oct 30 22:43:25.194554 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:25.194525 5748 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mh4rx/perf-node-gather-daemonset-qzh8d" Oct 30 22:43:25.233025 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:25.232998 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b287c83b-aa37-4063-9782-b62b202832d5/kube-rbac-proxy-web/0.log" Oct 30 22:43:25.653197 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:25.653165 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b287c83b-aa37-4063-9782-b62b202832d5/kube-rbac-proxy/0.log" Oct 30 22:43:26.078446 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:26.078368 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b287c83b-aa37-4063-9782-b62b202832d5/kube-rbac-proxy-thanos/0.log" Oct 30 22:43:26.503496 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:26.503462 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b287c83b-aa37-4063-9782-b62b202832d5/init-config-reloader/0.log" Oct 30 22:43:26.952417 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:26.952387 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-fglqz_a9ca84f8-a6d5-45f4-9137-799e05b12c11/prometheus-operator/0.log" Oct 30 22:43:27.381702 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:27.381630 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-fglqz_a9ca84f8-a6d5-45f4-9137-799e05b12c11/kube-rbac-proxy/0.log" Oct 30 22:43:27.835199 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:27.835168 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-7qc5s_b282542e-5b91-430d-bd5e-03af2e875952/prometheus-operator-admission-webhook/0.log" Oct 30 22:43:28.280701 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:28.280660 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7c58f6d48-ls9th_d8f2a8e7-53db-4867-bd38-a8d45b15e40d/telemeter-client/0.log" Oct 30 22:43:28.703091 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:28.703061 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7c58f6d48-ls9th_d8f2a8e7-53db-4867-bd38-a8d45b15e40d/reload/0.log" Oct 30 22:43:29.133944 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:29.133861 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7c58f6d48-ls9th_d8f2a8e7-53db-4867-bd38-a8d45b15e40d/kube-rbac-proxy/0.log" Oct 30 22:43:29.584183 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:29.584143 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76bbc756f6-s5wtw_a597eced-76f6-49e8-8176-2e9332188be3/thanos-query/0.log" Oct 30 22:43:30.022042 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:30.021975 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76bbc756f6-s5wtw_a597eced-76f6-49e8-8176-2e9332188be3/kube-rbac-proxy-web/0.log" Oct 30 22:43:30.143715 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:43:30.143679 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:43:30.454619 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:30.454593 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76bbc756f6-s5wtw_a597eced-76f6-49e8-8176-2e9332188be3/kube-rbac-proxy/0.log" Oct 30 22:43:30.920080 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:30.920053 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76bbc756f6-s5wtw_a597eced-76f6-49e8-8176-2e9332188be3/prom-label-proxy/0.log" Oct 30 22:43:31.338925 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:31.338835 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76bbc756f6-s5wtw_a597eced-76f6-49e8-8176-2e9332188be3/kube-rbac-proxy-rules/0.log" Oct 30 22:43:31.777196 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:31.777164 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76bbc756f6-s5wtw_a597eced-76f6-49e8-8176-2e9332188be3/kube-rbac-proxy-metrics/0.log" Oct 30 22:43:37.094529 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:37.094497 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j4v8r_3df036ff-1c00-4ce2-9e86-8acbeedf6c82/ovn-acl-logging/0.log" Oct 30 22:43:37.118410 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:37.118384 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j4v8r_3df036ff-1c00-4ce2-9e86-8acbeedf6c82/ovn-acl-logging/0.log" Oct 30 22:43:39.078916 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:39.078868 5748 scope.go:122] "RemoveContainer" containerID="5b795d09a0e327064ab0815e1f55f79ada8d6b340595449a138d1f78b53e69f4" Oct 30 22:43:40.448014 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:40.447972 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-rq2vl_2dbd490f-d94c-4f12-9106-a12117deda9d/download-server/0.log" Oct 30 22:43:43.691086 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:43.691047 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t7djv"] Oct 30 22:43:43.691534 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:43.691274 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t7djv" podUID="818838b2-f54f-4591-b855-73014d18a06e" containerName="istio-proxy" containerID="cri-o://8e8d2c844b854181452bd532cbedb6b871452df3fe11170cebeb2c75ee457588" gracePeriod=30 Oct 30 22:43:43.741794 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:43.741755 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-98fnd"] Oct 30 22:43:43.742016 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:43.741990 5748 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-98fnd" podUID="d1c1a2b1-f312-47b1-b437-2d8f1760eea0" containerName="istio-proxy" containerID="cri-o://a01f6e2e7773fa3f06e4d5a14a79b1ecb65ff4655a070fd2bd7e3357681df9f1" gracePeriod=30 Oct 30 22:43:49.027369 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.027339 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t7djv" Oct 30 22:43:49.030154 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.030138 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-98fnd" Oct 30 22:43:49.075270 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.075245 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/818838b2-f54f-4591-b855-73014d18a06e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/818838b2-f54f-4591-b855-73014d18a06e-istio-envoy\") pod \"818838b2-f54f-4591-b855-73014d18a06e\" (UID: \"818838b2-f54f-4591-b855-73014d18a06e\") " Oct 30 22:43:49.075391 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.075281 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-istio-podinfo\") pod \"d1c1a2b1-f312-47b1-b437-2d8f1760eea0\" (UID: \"d1c1a2b1-f312-47b1-b437-2d8f1760eea0\") " Oct 30 22:43:49.075391 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.075341 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-workload-socket\") pod \"d1c1a2b1-f312-47b1-b437-2d8f1760eea0\" (UID: \"d1c1a2b1-f312-47b1-b437-2d8f1760eea0\") " Oct 30 22:43:49.075522 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.075389 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-workload-certs\") pod \"d1c1a2b1-f312-47b1-b437-2d8f1760eea0\" (UID: \"d1c1a2b1-f312-47b1-b437-2d8f1760eea0\") " Oct 30 22:43:49.075522 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.075428 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/818838b2-f54f-4591-b855-73014d18a06e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/818838b2-f54f-4591-b855-73014d18a06e-istio-podinfo\") pod \"818838b2-f54f-4591-b855-73014d18a06e\" (UID: \"818838b2-f54f-4591-b855-73014d18a06e\") " Oct 30 22:43:49.075522 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.075462 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-istiod-ca-cert\") pod \"d1c1a2b1-f312-47b1-b437-2d8f1760eea0\" (UID: \"d1c1a2b1-f312-47b1-b437-2d8f1760eea0\") " Oct 30 22:43:49.075522 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.075506 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/818838b2-f54f-4591-b855-73014d18a06e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/818838b2-f54f-4591-b855-73014d18a06e-credential-socket\") pod \"818838b2-f54f-4591-b855-73014d18a06e\" (UID: \"818838b2-f54f-4591-b855-73014d18a06e\") " Oct 30 22:43:49.075740 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.075672 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-workload-socket" pod "d1c1a2b1-f312-47b1-b437-2d8f1760eea0" (UID: "d1c1a2b1-f312-47b1-b437-2d8f1760eea0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:43:49.075740 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.075679 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/818838b2-f54f-4591-b855-73014d18a06e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/818838b2-f54f-4591-b855-73014d18a06e-istiod-ca-cert\") pod \"818838b2-f54f-4591-b855-73014d18a06e\" (UID: \"818838b2-f54f-4591-b855-73014d18a06e\") " Oct 30 22:43:49.075860 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.075764 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-istio-token\" (UniqueName: \"kubernetes.io/projected/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-istio-token\") pod \"d1c1a2b1-f312-47b1-b437-2d8f1760eea0\" (UID: \"d1c1a2b1-f312-47b1-b437-2d8f1760eea0\") " Oct 30 22:43:49.075860 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.075761 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-workload-certs" pod "d1c1a2b1-f312-47b1-b437-2d8f1760eea0" (UID: "d1c1a2b1-f312-47b1-b437-2d8f1760eea0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:43:49.075860 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.075780 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/818838b2-f54f-4591-b855-73014d18a06e-credential-socket" pod "818838b2-f54f-4591-b855-73014d18a06e" (UID: "818838b2-f54f-4591-b855-73014d18a06e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:43:49.075860 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.075797 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-istio-data\") pod \"d1c1a2b1-f312-47b1-b437-2d8f1760eea0\" (UID: \"d1c1a2b1-f312-47b1-b437-2d8f1760eea0\") " Oct 30 22:43:49.075860 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.075830 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-credential-socket\") pod \"d1c1a2b1-f312-47b1-b437-2d8f1760eea0\" (UID: \"d1c1a2b1-f312-47b1-b437-2d8f1760eea0\") " Oct 30 22:43:49.076170 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.075861 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-istio-envoy\") pod \"d1c1a2b1-f312-47b1-b437-2d8f1760eea0\" (UID: \"d1c1a2b1-f312-47b1-b437-2d8f1760eea0\") " Oct 30 22:43:49.076170 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.075948 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/818838b2-f54f-4591-b855-73014d18a06e-istio-token\" (UniqueName: \"kubernetes.io/projected/818838b2-f54f-4591-b855-73014d18a06e-istio-token\") pod \"818838b2-f54f-4591-b855-73014d18a06e\" (UID: \"818838b2-f54f-4591-b855-73014d18a06e\") " Oct 30 22:43:49.076170 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.075975 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-istiod-ca-cert" pod "d1c1a2b1-f312-47b1-b437-2d8f1760eea0" (UID: "d1c1a2b1-f312-47b1-b437-2d8f1760eea0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:43:49.076170 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.075986 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/818838b2-f54f-4591-b855-73014d18a06e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/818838b2-f54f-4591-b855-73014d18a06e-istio-data\") pod \"818838b2-f54f-4591-b855-73014d18a06e\" (UID: \"818838b2-f54f-4591-b855-73014d18a06e\") " Oct 30 22:43:49.076170 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.076065 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-kube-api-access-6lv7l\" (UniqueName: \"kubernetes.io/projected/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-kube-api-access-6lv7l\") pod \"d1c1a2b1-f312-47b1-b437-2d8f1760eea0\" (UID: \"d1c1a2b1-f312-47b1-b437-2d8f1760eea0\") " Oct 30 22:43:49.076170 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.076101 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/818838b2-f54f-4591-b855-73014d18a06e-kube-api-access-bhw6f\" (UniqueName: \"kubernetes.io/projected/818838b2-f54f-4591-b855-73014d18a06e-kube-api-access-bhw6f\") pod \"818838b2-f54f-4591-b855-73014d18a06e\" (UID: \"818838b2-f54f-4591-b855-73014d18a06e\") " Oct 30 22:43:49.076514 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.076149 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/818838b2-f54f-4591-b855-73014d18a06e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/818838b2-f54f-4591-b855-73014d18a06e-workload-socket\") pod \"818838b2-f54f-4591-b855-73014d18a06e\" (UID: \"818838b2-f54f-4591-b855-73014d18a06e\") " Oct 30 22:43:49.076514 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.076230 5748 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/818838b2-f54f-4591-b855-73014d18a06e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/818838b2-f54f-4591-b855-73014d18a06e-workload-certs\") pod \"818838b2-f54f-4591-b855-73014d18a06e\" (UID: \"818838b2-f54f-4591-b855-73014d18a06e\") " Oct 30 22:43:49.076640 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.076596 5748 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-workload-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:43:49.076640 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.076619 5748 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-workload-certs\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:43:49.076640 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.076636 5748 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-istiod-ca-cert\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:43:49.076803 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.076654 5748 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/818838b2-f54f-4591-b855-73014d18a06e-credential-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:43:49.076803 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.076101 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-istio-data" pod "d1c1a2b1-f312-47b1-b437-2d8f1760eea0" (UID: "d1c1a2b1-f312-47b1-b437-2d8f1760eea0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:43:49.076803 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.076127 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/818838b2-f54f-4591-b855-73014d18a06e-istiod-ca-cert" pod "818838b2-f54f-4591-b855-73014d18a06e" (UID: "818838b2-f54f-4591-b855-73014d18a06e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:43:49.076803 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.076175 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/818838b2-f54f-4591-b855-73014d18a06e-istio-data" pod "818838b2-f54f-4591-b855-73014d18a06e" (UID: "818838b2-f54f-4591-b855-73014d18a06e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:43:49.077067 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.076886 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/818838b2-f54f-4591-b855-73014d18a06e-workload-certs" pod "818838b2-f54f-4591-b855-73014d18a06e" (UID: "818838b2-f54f-4591-b855-73014d18a06e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:43:49.077129 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.077112 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-credential-socket" pod "d1c1a2b1-f312-47b1-b437-2d8f1760eea0" (UID: "d1c1a2b1-f312-47b1-b437-2d8f1760eea0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:43:49.077306 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.077272 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/818838b2-f54f-4591-b855-73014d18a06e-workload-socket" pod "818838b2-f54f-4591-b855-73014d18a06e" (UID: "818838b2-f54f-4591-b855-73014d18a06e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:43:49.077984 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.077952 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-istio-podinfo" pod "d1c1a2b1-f312-47b1-b437-2d8f1760eea0" (UID: "d1c1a2b1-f312-47b1-b437-2d8f1760eea0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:43:49.078132 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.078108 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/818838b2-f54f-4591-b855-73014d18a06e-istio-podinfo" pod "818838b2-f54f-4591-b855-73014d18a06e" (UID: "818838b2-f54f-4591-b855-73014d18a06e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:43:49.078382 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.078352 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-istio-token" pod "d1c1a2b1-f312-47b1-b437-2d8f1760eea0" (UID: "d1c1a2b1-f312-47b1-b437-2d8f1760eea0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:43:49.078500 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.078478 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-kube-api-access-6lv7l" pod "d1c1a2b1-f312-47b1-b437-2d8f1760eea0" (UID: "d1c1a2b1-f312-47b1-b437-2d8f1760eea0"). InnerVolumeSpecName "kube-api-access-6lv7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:43:49.078944 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.078922 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/818838b2-f54f-4591-b855-73014d18a06e-istio-token" pod "818838b2-f54f-4591-b855-73014d18a06e" (UID: "818838b2-f54f-4591-b855-73014d18a06e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:43:49.079376 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.079360 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-istio-envoy" pod "d1c1a2b1-f312-47b1-b437-2d8f1760eea0" (UID: "d1c1a2b1-f312-47b1-b437-2d8f1760eea0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:43:49.079450 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.079373 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/818838b2-f54f-4591-b855-73014d18a06e-kube-api-access-bhw6f" pod "818838b2-f54f-4591-b855-73014d18a06e" (UID: "818838b2-f54f-4591-b855-73014d18a06e"). InnerVolumeSpecName "kube-api-access-bhw6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:43:49.079595 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.079578 5748 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/818838b2-f54f-4591-b855-73014d18a06e-istio-envoy" pod "818838b2-f54f-4591-b855-73014d18a06e" (UID: "818838b2-f54f-4591-b855-73014d18a06e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:43:49.177632 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.177608 5748 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/818838b2-f54f-4591-b855-73014d18a06e-workload-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:43:49.177632 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.177630 5748 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/818838b2-f54f-4591-b855-73014d18a06e-workload-certs\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:43:49.177786 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.177640 5748 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/818838b2-f54f-4591-b855-73014d18a06e-istio-envoy\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:43:49.177786 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.177650 5748 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-istio-podinfo\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:43:49.177786 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.177659 5748 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/818838b2-f54f-4591-b855-73014d18a06e-istio-podinfo\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:43:49.177786 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.177669 5748 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/818838b2-f54f-4591-b855-73014d18a06e-istiod-ca-cert\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:43:49.177786 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.177678 5748 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-istio-token\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:43:49.177786 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.177686 5748 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-istio-data\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:43:49.177786 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.177694 5748 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-credential-socket\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:43:49.177786 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.177704 5748 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-istio-envoy\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:43:49.177786 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.177713 5748 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/818838b2-f54f-4591-b855-73014d18a06e-istio-token\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:43:49.177786 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.177721 5748 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/818838b2-f54f-4591-b855-73014d18a06e-istio-data\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:43:49.177786 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.177729 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lv7l\" (UniqueName: \"kubernetes.io/projected/d1c1a2b1-f312-47b1-b437-2d8f1760eea0-kube-api-access-6lv7l\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:43:49.177786 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.177737 5748 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhw6f\" (UniqueName: \"kubernetes.io/projected/818838b2-f54f-4591-b855-73014d18a06e-kube-api-access-bhw6f\") on node \"ip-10-0-143-61.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:43:49.325143 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.325054 5748 generic.go:356] "Generic (PLEG): container finished" podID="818838b2-f54f-4591-b855-73014d18a06e" containerID="8e8d2c844b854181452bd532cbedb6b871452df3fe11170cebeb2c75ee457588" exitCode=0 Oct 30 22:43:49.325143 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.325101 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t7djv" Oct 30 22:43:49.325386 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.325149 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t7djv" event={"ID":"818838b2-f54f-4591-b855-73014d18a06e","Type":"ContainerDied","Data":"8e8d2c844b854181452bd532cbedb6b871452df3fe11170cebeb2c75ee457588"} Oct 30 22:43:49.325386 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.325197 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t7djv" event={"ID":"818838b2-f54f-4591-b855-73014d18a06e","Type":"ContainerDied","Data":"ce17533b0c3bb8d210083d87ee4da97b7b99f915e8dd5ba8a3cd38bdfbcc2865"} Oct 30 22:43:49.325386 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.325211 5748 scope.go:122] "RemoveContainer" containerID="8e8d2c844b854181452bd532cbedb6b871452df3fe11170cebeb2c75ee457588" Oct 30 22:43:49.330076 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.330047 5748 generic.go:356] "Generic (PLEG): container finished" podID="d1c1a2b1-f312-47b1-b437-2d8f1760eea0" containerID="a01f6e2e7773fa3f06e4d5a14a79b1ecb65ff4655a070fd2bd7e3357681df9f1" exitCode=0 Oct 30 22:43:49.330190 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.330155 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-98fnd" event={"ID":"d1c1a2b1-f312-47b1-b437-2d8f1760eea0","Type":"ContainerDied","Data":"a01f6e2e7773fa3f06e4d5a14a79b1ecb65ff4655a070fd2bd7e3357681df9f1"} Oct 30 22:43:49.330249 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.330202 5748 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-98fnd" event={"ID":"d1c1a2b1-f312-47b1-b437-2d8f1760eea0","Type":"ContainerDied","Data":"31e3196705f7a08f4eeba9bbb51936f872896eda43a21afbf3346d580c2fcb74"} Oct 30 22:43:49.330302 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.330276 5748 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-98fnd" Oct 30 22:43:49.339493 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.339475 5748 scope.go:122] "RemoveContainer" containerID="8e8d2c844b854181452bd532cbedb6b871452df3fe11170cebeb2c75ee457588" Oct 30 22:43:49.339778 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:43:49.339756 5748 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e8d2c844b854181452bd532cbedb6b871452df3fe11170cebeb2c75ee457588\": container with ID starting with 8e8d2c844b854181452bd532cbedb6b871452df3fe11170cebeb2c75ee457588 not found: ID does not exist" containerID="8e8d2c844b854181452bd532cbedb6b871452df3fe11170cebeb2c75ee457588" Oct 30 22:43:49.339857 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.339787 5748 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e8d2c844b854181452bd532cbedb6b871452df3fe11170cebeb2c75ee457588"} err="failed to get container status \"8e8d2c844b854181452bd532cbedb6b871452df3fe11170cebeb2c75ee457588\": rpc error: code = NotFound desc = could not find container \"8e8d2c844b854181452bd532cbedb6b871452df3fe11170cebeb2c75ee457588\": container with ID starting with 8e8d2c844b854181452bd532cbedb6b871452df3fe11170cebeb2c75ee457588 not found: ID does not exist" Oct 30 22:43:49.339857 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.339806 5748 scope.go:122] "RemoveContainer" containerID="a01f6e2e7773fa3f06e4d5a14a79b1ecb65ff4655a070fd2bd7e3357681df9f1" Oct 30 22:43:49.349774 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.349734 5748 scope.go:122] "RemoveContainer" containerID="a01f6e2e7773fa3f06e4d5a14a79b1ecb65ff4655a070fd2bd7e3357681df9f1" Oct 30 22:43:49.350122 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:43:49.350092 5748 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a01f6e2e7773fa3f06e4d5a14a79b1ecb65ff4655a070fd2bd7e3357681df9f1\": container with ID starting with a01f6e2e7773fa3f06e4d5a14a79b1ecb65ff4655a070fd2bd7e3357681df9f1 not found: ID does not exist" containerID="a01f6e2e7773fa3f06e4d5a14a79b1ecb65ff4655a070fd2bd7e3357681df9f1" Oct 30 22:43:49.350212 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.350131 5748 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a01f6e2e7773fa3f06e4d5a14a79b1ecb65ff4655a070fd2bd7e3357681df9f1"} err="failed to get container status \"a01f6e2e7773fa3f06e4d5a14a79b1ecb65ff4655a070fd2bd7e3357681df9f1\": rpc error: code = NotFound desc = could not find container \"a01f6e2e7773fa3f06e4d5a14a79b1ecb65ff4655a070fd2bd7e3357681df9f1\": container with ID starting with a01f6e2e7773fa3f06e4d5a14a79b1ecb65ff4655a070fd2bd7e3357681df9f1 not found: ID does not exist" Oct 30 22:43:49.497227 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.497195 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t7djv"] Oct 30 22:43:49.505097 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.505068 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t7djv"] Oct 30 22:43:49.620243 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.620167 5748 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-98fnd"] Oct 30 22:43:49.625109 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:49.625081 5748 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-98fnd"] Oct 30 22:43:50.361691 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:50.361662 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nrm49_c2b5bbd1-4e37-4d49-9e28-bdb3c3da309f/dns/0.log" Oct 30 22:43:50.416412 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:50.416382 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="818838b2-f54f-4591-b855-73014d18a06e" path="/var/lib/kubelet/pods/818838b2-f54f-4591-b855-73014d18a06e/volumes" Oct 30 22:43:50.416816 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:50.416802 5748 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1c1a2b1-f312-47b1-b437-2d8f1760eea0" path="/var/lib/kubelet/pods/d1c1a2b1-f312-47b1-b437-2d8f1760eea0/volumes" Oct 30 22:43:50.788292 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:50.788254 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nrm49_c2b5bbd1-4e37-4d49-9e28-bdb3c3da309f/kube-rbac-proxy/0.log" Oct 30 22:43:51.208793 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:51.208763 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8rbsm_d9f7709e-2664-452f-b70b-f4649675781e/dns-node-resolver/0.log" Oct 30 22:43:55.323625 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:43:55.323594 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8vqf9_59571898-f15b-4b8f-b043-0e5af369c39e/node-ca/0.log" Oct 30 22:44:00.143437 ip-10-0-143-61 kubenswrapper[5748]: E1030 22:44:00.143391 5748 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vfrjv" containerName="extractor" Oct 30 22:44:03.507830 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:44:03.507797 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6bcd447999-f4qch_e692bffa-b34f-4ce9-9db8-6667ed013afb/kube-auth-proxy/0.log" Oct 30 22:44:03.934918 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:44:03.934865 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-948nx_f411f535-b1bc-408b-afb9-3299fc9d5bd6/istio-proxy/0.log" Oct 30 22:44:10.157839 ip-10-0-143-61 kubenswrapper[5748]: I1030 22:44:10.157793 5748 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zb9t2_74930384-19fd-4c0b-b38f-b2f895f7e6e8/serve-healthcheck-canary/0.log"